Hints, tips, and tricks for debugging IPFS.

[autonome]

Developers run into problems with IPFS, and we should have resources to help them.

Whether it's a one-page debugging cheat sheet or a series of pages covering specific topics, I'm filing this as a scratchpad to drop instances of valuable debugging information.

Ideally there'd be a sequential checklist for common scenarios like content availability and node connectivity.

[autonome]

Why is IPFS running slowly?

https://stackoverflow.com/questions/66450619/how-do-i-find-out-why-ipfs-is-running-slowly

@aschmahmann answers:

Commands like ipfs ls are going to the network to try and find data. If for some reason that data is not findable (e.g. nobody online has it, there is one node that has the data but it's behind a NAT, the node that has it has not yet advertised the data in a way that your node can find it, etc.) then go-ipfs will just keeping looking for who has the data.

You can take a look at what's going on with Bitswap (e.g. ipfs bitswap stat) and that can help you determine if you're stuck looking for data (i.e. data in the wantlist).

Some functions also have flags like --stream or --progress that can help you see incremental updates. For logging behavior there is ipfs log, where ipfs log level can help you inspect subsystems further.

Note you can pass a timeout flag to basically all go-ipfs commands (see ipfs --help for more info) if you're concerned about your CLI not responding quickly enough when the data just isn't available on the network.

[autonome]

Steps @aschmahmann used in debugging session with a collab, to figure out why a CID was not fetchable, walking through seeing if peer was dialable, etc.

ipfs id // list my node addresses

// check connectable with public tools, eg:
https://portchecker.co/

// with peer id of node you're trying to reach
// get its addresses
// try to connect to it
ipfs id {PeerID}

try a bunch of connection methods
ipfs swarm connect /p2p/{PeerID}

ipfs swarm connect /ip4/{IPAddress}/tcp/4001/p2p/{PeerID}

ipfs swarm connect /ip4/{IPAddress}/udp/4001/quic/p2p/{PeerID}

// list who has this content
ipfs dht findprovs {CID}

[autonome]

Some tips here: https://github.com/ipfs/go-ipfs/blob/master/docs/debug-guide.md

[autonome]

@aschmahmann answering a "why is my ipfs slow" question w/ some practical tips: https://stackoverflow.com/questions/66450619/how-do-i-find-out-why-ipfs-is-running-slowly

[autonome]

From @aschmahmann, who will someday write the best-selling ebook on this topic:

Step 1 to debugging is always try ipfs swarm connect (found from ipfs id on the target node)

some highlights to consider during debugging here:

Gateways can be misleading, sometimes you'll just happen to be connected to a gateway and so it'll pull the content from you even if you're unreachable (e.g. NAT'd)

If your peer ID isn't findable via ipfs dht findprovs then you're leaving things up to chance, and if you're not running a DHT server that chance is basically zero

If this is the problem then for debugging try ipfs dht provide and see if things work

If so then there's a provider record problem and we can discuss options. Hopefully go-ipfs v0.9.0 with the experimental accelerated DHT client will make this largely go away

Your peer needs to be dialable (ipfs swarm connect )
It should be findable via the dht (i.e. after restarting your node so you're not already connected doing ipfs swarm connect /p2p/peerID )

Gateways can be misleading, sometimes you'll just happen to be connected to a gateway and so it'll pull the content from you even if you're unreachable (e.g. NAT'd)
What's amazing + confusing here is the magic of p2p systems + content addressing. The data can flow from NAT'd node -> gateway -> friend who requested it and the friend can advertise the data in the DHT and everything will appear totally fine even though nobody is fetching content from the original host node anymore.
Essentially the resiliency of the system and how data can flow through any number of avenues makes these thing hard to debug from a UX layer (e.g. "hey how come sometimes my file shows up and sometimes it doesn't") and instead you have to be a bit more precise about what "working means" for you. For most people that's ipfs dht findprovs giving back their PeerID and ipfs swarm connect /p2p/MyPeerID working.

[atopal]

Marked as p1, since this is such a common issue and we currently have no good way to address it.

[TheDiscordian]

This comes up fairly often:

Q: Why is my VPS accusing me of scanning their internal network

A:
By default go-ipfs utilises mDNS which results in it trying to talk to a lot of internal network IPs. You can resolve the issue by using a server profile by executing ipfs config profile apply server on the node.

[johnnymatthews]

Tasks

• Create a Troubleshooting section under How-tos.
• Create a page called for each of the topics discussed in this issue under Troubleshooting.
  • Grammar and spellcheck the content, and flesh-out where necessary.
• Add sidebar nav items for each page.

[alexmmueller]

Hey @johnnymatthews! I will take care of this issue, please assign it to me. Thank you!

[johnnymatthews]

Perfect, thanks for jumping on this one @alexmmueller. If you've got any questions drop them in here - you know the drill.

[johnnymatthews]

Hey @alexmmueller, how's your progress on this?