Two nodes in the same machine - can't connect from one to another

[sidaudhi]

Issue and Steps to Reproduce

I have two nodes running on ports 9735 and 9112 on the same DigitalOcean droplet.

When I try to connect from one to the other:
docker exec lightning lightning-cli connect 038de60a0cb43eea67e0bcb6e207daebad312a245cedd8b09deb2c5ba4bb19f601 138.68.231.196 9112

i get the error:
{ "code" : -1, "message" : "138.68.231.196:9112: Cryptographic handshake: peer closed connection (wrong key?). " }

Both my nodes are built out of image: elementsproject/lightningd and ports 9735 and 9112 are open through the DigitalOcean UFW firewall.

getinfo output

Node 1:

{
  "id": "0230d70b0506aa447a1db407cc04e41448ab92bbd373f5c9a1029a40f357fcdcad",
  "alias": "LN.pay",
  "color": "20b0cf",
  "num_peers": 2,
  "num_pending_channels": 0,
  "num_active_channels": 1,
  "num_inactive_channels": 1,
  "address": [
    {
      "type": "ipv4",
      "address": "138.68.231.196",
      "port": 9735
    }
  ],
  "binding": [
  ],
  "version": "v0.6.3rc1-118-g0da4054",
  "blockheight": 562550,
  "network": "bitcoin",
  "msatoshi_fees_collected": 0
}

Node 2:

{
  "id": "038de60a0cb43eea67e0bcb6e207daebad312a245cedd8b09deb2c5ba4bb19f601",
  "alias": "ln.pay.2",
  "color": "20b0cf",
  "num_peers": 0,
  "num_pending_channels": 0,
  "num_active_channels": 0,
  "num_inactive_channels": 0,
  "address": [
    {
      "type": "ipv4",
      "address": "138.68.231.196",
      "port": 9112
    }
  ],
  "binding": [
  ],
  "version": "v0.6.3rc1-118-g0da4054",
  "blockheight": 562550,
  "network": "bitcoin",
  "msatoshi_fees_collected": 0
}

Docker compose setup:

clightning_bitcoin:
    image: elementsproject/lightningd
    container_name: lightning
    command:
      - --announce-addr=138.68.231.196:9735
    environment:
      EXPOSE_TCP: "true"
    expose:
      - "9735"
    ports:
      - "0.0.0.0:9735:9735"

clightning_02_bitcoin:
    image: elementsproject/lightningd
    container_name: lightning_02
    command:
      - --announce-addr=138.68.231.196:9112
    environment:
      EXPOSE_TCP: "true"
    expose:
      - "9735"
    ports:
      - "0.0.0.0:9112:9735"

[cdecker]

Can you attempt to connect from clightning_bitcoin to clightning_02_bitcoin? You'll still be able to set up a channel in the opposite direction. I'm just wondering whether the connect syntax is correct and that the port is not being interpreted as something else.

You can also use the lightning-cli connect [node_id]@[node_ip]:[port] syntax.

[sidaudhi]

I have tried the connection both ways from clightning_bitcoin and from clightning_02_bitcoin. Neither works.

The syntax didnt matter. Tried both of these and they have the same error.

docker exec lightning lightning-cli connect 038de60a0cb43eea67e0bcb6e207daebad312a245cedd8b09deb2c5ba4bb19f601@138.68.231.196:9112
docker exec lightning_02 lightning-cli connect 0230d70b0506aa447a1db407cc04e41448ab92bbd373f5c9a1029a40f357fcdcad@138.68.231.196:9735

Is there anything else you'd suggest I try to do to get to the root cause?

Do you see any issues with the docker-compose ports and expose

[cdecker]

expose doesn't really do a lot. According to the docs they are not used from the host which is the path you are trying to access the nodes from:

Expose ports without publishing them to the host machine - they’ll only be accessible to linked services. Only the internal port can be specified.

ports look fine to me, it shouldn't matter that c-lightning is being told to listen on port 9735, but 9112 from the host is being forwarded to that port, c-lightning can deal with this kind of mappings.

[cdecker]

Another alternative would be to try and connect directly over the docker bridge. If you run docker exec lightning_02 ip addr you'll probably see a 172.x.y.z address (that's attached to the bridge interface). You can try connecting to that instead.

If that doesn't work I'd suggest taking out telnet and try a simple TCP connection before going with the full handshake.

[sidaudhi]

This was helpful. But the problem now seems deeper and similar to #1798

The difference is that the ports are open on the DigitalOcean droplet, yet any telnet / netcat call to the IP and port is throwing an error:

netcat: connect to 138.68.231.196 port 9735 (tcp) failed: Connection refused

When I check other applications running on the machine on ports 8080 and 8081, the netcat / telnet is successful:

Connection to 138.68.231.196 8081 port [tcp/tproxy] succeeded!

lsof TCP output

The firewall is good:

[sidaudhi]

Solved. The problem was with the --bind-addr flag not being present.

The moment i added: --bind-addr=0.0.0.0:9735 to the command section of the container, I was able to telnet to the port 9735 successfully.

Now I am able to connect between the nodes and from outside the VM as well and create inbound channels to my nodes.

Does it make sense to add this to the README.md so that in future inbound connections aren't an issue?

[cdecker]

@rustyrussell does the --announce-addr have any influence on the address/port we bind to?

[niftynei]

I've run into this before. If you set --announce-addr you also have to specify a --bind-addr, as @sidaudhi discovered.

[sidaudhi]

I create a pull request for a README.md update so that this issue doesn't come up in the future