Content can sometimes only be found after hours or days

[Stebalien]

See: #6382 (comment)

This isn't the first time I've seen this and it warrants investigation. There used to be an issue where bitswap would time out too quickly sending out provider records for new blocks. However, we now provide in a background queue.

It could be that this background queue is getting backed up.

[warlockdn]

@Stebalien is this the same issue that you are facing ?

I am adding a file and then pinning the hash.

ipfs add -r <filename>
ipfs pin add <hash>
https://ipfs.io/ipfs/<hash>

File doesn't seem to be available.

If this is the case. Then please make a page of disclaimer like information where you guys tell about the process and when will it be available. Its beating the purpose of using the system.

Please advise

[warlockdn]

any ideas ? how to make this work ???

[hacdias]

There are more people having this issue (#6425). We also have a report on IPFS Desktop (ipfs/ipfs-desktop#955) and @olizilla and I have not been able to get our files through any public gateway.

@Stebalien could you confirm if this is only happening after 0.4.21?

[miguelito9]

Regarding my problem report #955, it has been happening since new Win desktop ver 0.7.2 install on 23 May 19. Thought it might be something about my external/USB install or setup but didn't find anything. Then made problem report apparently in wrong place 15 days ago at https://discuss.ipfs.io/t/difficulty-accessing-ipfs-desktop-saved-files-on-external-hd/5503.

[hacdias]

@miguelito9 weird, 0.7.2 still uses go-ipfs 0.4.19. You shouldn't have any issues with that I think... I'll investigate it better

[miguelito9]

Win Desktop app status screen states: VERSION go-ipfs v0.4.21 . Don't know what it was at time of install.

[miguelito9]

I am seeing block activity averaging about 0.3 Mb/s with a couple of spikes to about 0.9 in my node over the last day with a small increase in storage used. Still getting 504 gateway time-outs when trying to access with browser (no ipfs companion) the small png file I uploaded to my node at
https://ipfs.io/ipfs/QmdMkoevxiyexeM21jAgKDrjrZ6G8DqvYpUBs6s3nVNeab

[Stebalien]

We don't believe this has anything to do with the local IPFS version.

[miguelito9]

We don't believe this has anything to do with the local IPFS version.

Okay. Any thoughts on possible other causes?

[Stebalien]

Two possible issues:

1. Content Resolution And Gateway Performance #6383
2. We've noticed that the gateways are bottle necking on finding content (we severely limit the number of parallel content finding workers as they can impact performance). We're testing out a patch (WIP improve provider request rate limiting go-bitswap#138) on the gateways to see if this is the case.

[beenotung]

I remember in the early day, (not remember which version) I would publish content then use ipfs.io gateway to see the content without having a public IP.
But in the current version (0.4.20 or alike, I need to check when I back to home), the content is not accessible from ipfs.io if my node doesn't have a public IP.

[Stebalien]

That was because:

1. The DHT would dial pretty much every node on the network due to a bug. This bug has partially been fixed.
2. The network was small enough that every node could be connected to every other node, all the time. That's no longer the case.

[beenotung]

Would it be practical if most of the nodes without public ip can connect to some common nodes, e.g. ipfs.io, so those common nodes can rely the connections between the nodes without public ip and the rest of the network?

[beenotung]

As long as we're relying on TCP/IP, we need some nodes to act like router/relay to bridge the network, unless we're using protocol like NDN that doesn't rely on host address when sending packets.

[Stebalien]

Would it be practical if most of the nodes without public ip can connect to some common nodes, e.g. ipfs.io, so those common nodes can rely the connections between the nodes without public ip and the rest of the network?

We have relays for this but we had to significantly reduce our relay deployment due to costs. There's a lot of traffic on the IPFS network and relaying all of it was unsustainable.

However, you can still use them by enabling autorelay: ipfs config --bool Swarm.EnableAutoRelay true.

---

We're also slowly working on better NAT traversal (e.g., ICE) but the plan has been to use relays to coordinate things like ICE.

[Kuurusch]

Maybe my ebservations helps: So I tried to share a 20GB-file between two Nodes (Desktop Computers). I've added the file on Node A and pined it. Afterwards I've downloaded it on Node B with get. It started well but got stuck after 657MB. I've figured out, that I've could solve the problem by directly connect the two Nodes over "ipfs swarm connect". So it seams that the file was after pinning uploaded only partially to ipfs and Node B was not able to find Node A and directly connect to Node A and download it from there. Seems to me that it should be important that "get" should try to find the source-Node, when it can't find the entire file in IPFS.

[lordcirth]

Maybe my ebservations helps: So I tried to share a 20GB-file between two Nodes (Desktop Computers). I've added the file on Node A and pined it. Afterwards I've downloaded it on Node B with get. It started well but got stuck after 657MB. I've figured out, that I've could solve the problem by directly connect the two Nodes over "ipfs swarm connect". So it seams that the file was after pinning uploaded only partially to ipfs and Node B was not able to find Node A and directly connect to Node A and download it from there. Seems to me that it should be important that "get" should try to find the source-Node, when it can't find the entire file in IPFS.

Thanks for your report. Were either of the desktops using Auto-relay? Was one or both of them behind a NAT, or were they on the same local network? Thanks.

[Kuurusch]

@lordcirth Yes, both computers where in separate subnets in a local network (University). One was connected over LAN and the other over WLAN.

[Stebalien]

See #6383.