Config Announce Address automatically Overwritten to /p2p/peer-id

[alecbcs]

Version information:

ipfs version 0.7.0

Description:

I've successfully gotten a couple of nodes to communicate in a private swarm through a relay by enabling both AutoRelay & RelayHop on one of the nodes with a publicly accessible connection, as well as also hard-coding the relay addresses for the other nodes in their announce config. This all works perfectly for ~15 minutes, until for some reason the announce addresses are overwritten on the nodes behind the NAT to /p2p/PEER-ID. After that point the nodes drop out of the swarm and become inaccessible by each other. Is there anyway to prevent the nodes from overwriting the manual announce address in the config?

[welcome]

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

• "Priority" labels will show how urgent this is for the team.
• "Status" labels will show if this is ready to be worked on, blocked, or in progress.
• "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

[Stebalien]

as well as also hard-coding the relay addresses for the other nodes in their announce config.

If you're hard-coding relay addresses, you shouldn't be using auto-relay. Auto relay means "autodetect whether or not I'm behind a NAT and, if so, try to find a relay and announce relay addresses".

Also note: At the moment, AutoRelay will use relays from a pre-defined list as randomly discovered relays on the network tended to behave very poorly. Ideally, we'd have some form of reputation/reliability score, but we haven't gotten around to implementing anything like this.

until for some reason the announce addresses are overwritten on the nodes behind the NAT to /p2p/PEER-ID.

This seems like a bug, regardless. What's the output of ipfs id?

[alecbcs]

If you're hard-coding relay addresses, you shouldn't be using auto-relay.

Thanks! This is really good to know, I'd incorrectly thought that without having auto-relay enabled the nodes wouldn't accept a relay connection at all.

Quick additional question, in doing some testing I have noticed that without auto-relay being enabled the nodes are still accessible if I look them up from the swarm boot strapper nodes, but they don't broadcast their announce address by default. Do you know why this would be? For example, if I view ipfs swarm addrs the addresses of the nodes behind the relay don't appear until after I run ipfs dht findpeer x.

This seems like a bug, regardless. What's the output of ipfs id?

For the first ~15 minutes the output of ipfs id is:

{
	"ID": "QmRohwX5DoX4QTqEqr6bUWPvVWYvhxwHN9YrSLHhebRm8M",
	"PublicKey": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC4625dp6WdmKuPywNuNjWDS3EufVTlFBYOQwnvxIkAqC5xeqXpSs2lkITKz7nkHpuVgYd95UzRzhwumysoTGGpx2fikMOD+fqh1NNlmfpNJ0qVhaZXizTH99q+t23R42bn99ogmCeAYx/RGIPFB9G7Re6d+Z4na54MKB/tm0gyTi4/sRcVW3STF4vM0OIatCtjXD/J8CNd5Ryg/RltcJG98nzs1hZulYI/gp95+kRQ5EqWkaWIv8ZqbzfKFv5ZN72XRNbd+6N6O9O9on5jbKEJGcnd1GvMPzhcRn4Qulg1Bakpq8jj4/Myj0PmbIwIs6g4otqFQH3bk9zERozo4KsdAgMBAAE=",
	"Addresses": [
		"/dns4/relay.arken.io/tcp/4001/p2p/12D3KooWL7hvR7nfQxAWMowgoWXWQwKEkQA8QPZrhKjateRTgcDm/p2p-circuit/p2p/QmRohwX5DoX4QTqEqr6bUWPvVWYvhxwHN9YrSLHhebRm8M/p2p/QmRohwX5DoX4QTqEqr6bUWPvVWYvhxwHN9YrSLHhebRm8M"
	],
	"AgentVersion": "go-ipfs/0.7.0/ea77213e3",
	"ProtocolVersion": "ipfs/0.1.0",
	"Protocols": [
		"/ipfs/bitswap",
		"/ipfs/bitswap/1.0.0",
		"/ipfs/bitswap/1.1.0",
		"/ipfs/bitswap/1.2.0",
		"/ipfs/id/1.0.0",
		"/ipfs/id/push/1.0.0",
		"/ipfs/lan/kad/1.0.0",
		"/ipfs/ping/1.0.0",
		"/libp2p/autonat/1.0.0",
		"/libp2p/circuit/relay/0.1.0",
		"/p2p/id/delta/1.0.0",
		"/x/"
	]
}

Then after ~15 minutes ipfs id changes to:

{
	"ID": "QmRohwX5DoX4QTqEqr6bUWPvVWYvhxwHN9YrSLHhebRm8M",
	"PublicKey": "CAASpgIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC4625dp6WdmKuPywNuNjWDS3EufVTlFBYOQwnvxIkAqC5xeqXpSs2lkITKz7nkHpuVgYd95UzRzhwumysoTGGpx2fikMOD+fqh1NNlmfpNJ0qVhaZXizTH99q+t23R42bn99ogmCeAYx/RGIPFB9G7Re6d+Z4na54MKB/tm0gyTi4/sRcVW3STF4vM0OIatCtjXD/J8CNd5Ryg/RltcJG98nzs1hZulYI/gp95+kRQ5EqWkaWIv8ZqbzfKFv5ZN72XRNbd+6N6O9O9on5jbKEJGcnd1GvMPzhcRn4Qulg1Bakpq8jj4/Myj0PmbIwIs6g4otqFQH3bk9zERozo4KsdAgMBAAE=",
	"Addresses": [
		"/p2p/QmRohwX5DoX4QTqEqr6bUWPvVWYvhxwHN9YrSLHhebRm8M"
	],
	"AgentVersion": "go-ipfs/0.7.0/ea77213e3",
	"ProtocolVersion": "ipfs/0.1.0",
	"Protocols": [
		"/ipfs/bitswap",
		"/ipfs/bitswap/1.0.0",
		"/ipfs/bitswap/1.1.0",
		"/ipfs/bitswap/1.2.0",
		"/ipfs/id/1.0.0",
		"/ipfs/id/push/1.0.0",
		"/ipfs/lan/kad/1.0.0",
		"/ipfs/ping/1.0.0",
		"/libp2p/autonat/1.0.0",
		"/libp2p/circuit/relay/0.1.0",
		"/p2p/id/delta/1.0.0",
		"/x/"
	]
}

[Stebalien]

Thanks! This is really good to know, I'd incorrectly thought that without having auto-relay enabled the nodes wouldn't accept a relay connection at all.

AutoRelay really just means "automatically find me a relay". In your case, just:

1. Make sure to connect to the relay you want to use (you can us peering.
2. Configure that relay in your announce list.

Do you know why this would be? For example, if I view ipfs swarm addrs the addresses of the nodes behind the relay don't appear until after I run ipfs dht findpeer x.

ipfs swarm addrs prints out all known addresses. You won't learn the addresses of a peer until you either connect to them or find them in the DHT.

but they don't broadcast their announce address by default.

Could you expand on this? If ipfs dht findpeer x works, the addresses have been broadcast (well, usually).

---

Regardless, announcing an "empty" address is definitely a bug that we'll have to track down.

[alecbcs]

Could you expand on this? If ipfs dht findpeer x works, the addresses have been broadcast (well, usually).

Is there any case that the nodes' addresses aren't passed further than the bootstrap nodes? What I'm seeing here is that as long as I and the peer behind the relay are directly connected to the relay node I'm able to lookup the routing information for the node. However it doesn't seem like the routing information of the node behind the relay is getting passed down to any of the other nodes. (As in running ipfs swarm addrs results in no address for the node behind the relay -- even on the swarm bootstrapper nodes -- until after I manually ping the node behind the relay.) Before when using AutoRelay the address of the node behind the relay seemed to get broadcast to all of the other nodes in the relatively small cluster so that running ipfs swarm addrs would result in showing the address of the relayed node even before manually making a connection to that node.

Right now it doesn't appear that this is having any effect on retrieving content from the cluster but I'm worried once there are more nodes than can connect to the relay peer at once that the peers behind the relay will become inaccessible.

Thank you for all your help diagnosing this issue @Stebalien!

[Stebalien]

Is there any case that the nodes' addresses aren't passed further than the bootstrap nodes?

As long as you have the DHT enabled, no. Your nodes should be pushing their addresses to the nodes closest to them in the DHT (as long as they're connected to at least one DHT node).

However, I think I may know what's wrong:

• How many bootstrap nodes do you have?
• How many publicly dialable nodes do you have?
• How have you configured your bootstrap node's DHTs? Are they defaulting to "dhtserver", or just "dht".

I ask because you may not actually be forming a functional DHT.

[alecbcs]

How many bootstrap nodes do you have?

2

How many publicly dialable nodes do you have?

7 (2 bootstrap nodes + 5 nodes publicly accessible through UPnP NATs) (Not counting any behind the relay.)

How have you configured your bootstrap node's DHTs? Are they defaulting to "dhtserver", or just "dht".

I've manually set all of the nodes to dhtserver.

[Stebalien]

I've manually set all of the nodes to dhtserver.

Don't do this for the nodes behind relays. We explicitly filter those out when querying as dialing DHT peers through relays is very slow (and unreliable).

7 (2 bootstrap nodes + 5 nodes publicly accessible through UPnP NATs) (Not counting any behind the relay.)

That should be enough.

---

In one of your unreachable relayed nodes, try running ipfs stats dht wan and look at the highest few buckets. The peers in those buckets are "closest" to this node and are responsible for storing this node's peer routing information.

Then go to one of those nodes and run ipfs swarm addrs to see what addresses that node thinks the node behind a relay has. That'll tell us if it knows the correct addresses.

If it does have the correct address, go to a node that's unable to connect to the node behind a relay and run ipfs dht query QmNodeBehindRelay. That will spit out a list of nodes "closest" to the target node, from this nodes perspective. That list should intersect with the list of nodes from the first step.

[aarshkshah1992]

@Stebalien @aschmahmann

I think I figured out the bug. There are two things at play here together:

1. AutoRelay wraps/overrides the configured Host AddressFactory with another AddressFactory that converts all public addresses to corresponding relay addresses. DNS addresses are treated as public addresses here. The code is here:

https://github.com/libp2p/go-libp2p/blob/fcf69647e945b30851fe394d140f7a70643e48cf/p2p/host/relay/autorelay.go#L70
https://github.com/libp2p/go-libp2p/blob/fcf69647e945b30851fe394d140f7a70643e48cf/p2p/host/relay/autorelay.go#L268

Now, since the address returned by the wrapped address factory(configured by Addresses.Announce here) will ONLY contain one address i.e. the public announce relay address, it will get filtered out by the cleanupAddressSet function here as it is a Relay address.

So, essentially, a call to host.Addrs() will return an empty address set.

---

2. Now, turns out that ipfs id uses the AddrInfoToP2pAddrs function in go-libp2p-core to get the peer multi-addresses to print. That function simply returns a /p2p/pid address if the peer has NO addresses which is why see the empty p2p address here.

---

I think the fix is to either disallow the user from using the EnableAutoRelay() option for non-Relay Servers if the Addresses.Announce option is set(we'd have to figure out how to do this) OR allow the AutoRelay address factory to return relay addresses as is without encapsulating them in another /p2p-circuit address as I don't see the harm that would cause.

Wdyt ?

[Stebalien]

Yeah, disallowing autorelay (client) in this case makes sense. That is, if RelayHop is not set and EnableAutoRelay and Addresses.Announce are set, we should fail on start. I can't think of a case where this would make sense.

Also, we should definitely detect the "no addresses" case in ipfs id and return no addresses.

[aarshkshah1992]

@Stebalien

IIUC, libp2p dosen't know that Addresses.Announce is set as that's done in IPFS. Whereas, EnableAutoRelay is a libp2p option. How would a libp2p node go about detecting this and fail ? Or do you mean to say we should do this in IPFS ?

Also, why did we change the AddrInfoToP2pAddrs function in libp2p core to return an empty p2p address rather than NO addresses if the peer's address set is empty ?

[Stebalien]

I believe this needs to happen in ipfs.

Also, why did we change the AddrInfoToP2pAddrs function in libp2p core to return an empty p2p address rather than NO addresses if the peer's address set is empty ?

Probably to fix a different bug? I can't remember.