Pin with ton of hashes won't start

[ghost]

Version information: go-ipfs/0.4.10/4679f80

Type: bug

Severity: High

Description:

ipfs pin add with a ton of hashes as input won't start doing anything. It'll just sit there, even with --progress it'll never move beyond `Fetched/Processed 0 nodes.

This applies to both cat hashes.txt | xargs ipfs pin add, as well as cat hashes.txt | ipfs pin add.

The list in question has 1199 items: pins.txt.

[whyrusleeping]

Interesting, we should look at what is holding it up. Does pinning just two hashes at a time work? I would be interested to see something like:

for i in `seq 2 1000`
do
    date
    echo $i hashes
    head -n$i hashes.txt | ipfs pin add 
done

[Stebalien]

This appears to be doing a lot of dialing.

[Stebalien]

Yeah, this looks like a dialing issue. I'm guessing it's a combination of:

1. Dial timeouts.
2. Dial backoffs.
3. Dial limits.

That is, we're queuing up enough dials such that, by the time we start processing each dial, it has already expired.

---

Using bitswap sessions for pin will significantly help but that's not a real fix. We should probably avoid infinitely queuing dials (have a finite dial queue and immediately reject or block all other dial requests).

[leerspace]

If anyone comes across this issue and wants a one-liner workaround, try cat hashes.txt | xargs -L 1 ipfs pin add. It's the same as the command in the first post which doesn't work, except it uses the -L 1 argument for xargs to force only one pin to be added at a time.

[ligi]

was this one solved with 0.4.13 and just forgotten to close? Unfortunately the workaround does not really work for me and I am stumbling into this one often with my kontinuum ci

[whyrusleeping]

@ligi how many hashes are you pinning at the same time?

[whyrusleeping]

@ligi also, is this with or without the progress feature?

[ligi]

@whyrusleeping no I do not use the progress feature - usure about the amount of hashes - I just try to pin my reports ..-)

[whyrusleeping]

I wonder if this is related to the pinning issue that @hsanjuan brought up recently.

@ligi when this happens, could you grab me a full stack trace of the daemon? (via curl localhost:5001/debug/pprof/goroutine\?debug=2)

[leerspace]

@whyrusleeping I think there might be a typo in that debug command. Based on this debug documentation it's singular "goroutine" instead of plural. The one in your post returns Unknown profile: goroutines for me.

This is the one from the debug document that seems to work for me:

curl localhost:5001/debug/pprof/goroutine\?debug=2

[whyrusleeping]

Whoops, my bad.

…

On Sat, Nov 18, 2017, 4:25 PM John Reed ***@***.***> wrote: @whyrusleeping <https://github.com/whyrusleeping> I think there might be a typo in that debug command. Based on this debug documentation <https://github.com/ipfs/go-ipfs/blob/master/docs/debug-guide.md> it's singular "goroutine" instead of plural. The one in your post returns Unknown profile: goroutines for me. This is the one from the debug document that seems to work for me: curl localhost:5001/debug/pprof/goroutine\?debug=2 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4122 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABL4HGsyYVaqOgQVv6kJJB3prqFn01xmks5s33VsgaJpZM4OumHC> .

[hsanjuan]

Yeah, some of my hanging tests are also pinning several things at the "same time" (sequentially but close in time) #4394

[hsanjuan]

Actually, thinking about it it's the opposite, while we cluster-pin several things at the same time ipfs-cluster does not trigger 2 ipfs pin requests at the same time. We queue them and only trigger the new one when the previous one is finished.

[Stebalien]

So, with #4403, this appears to work much better (secio/dialing change?). I now see my wantlist updating. However, IPFS appears to wait a while before actually pinning. Furthermore, it seems to start of very slowly, fetching each item on the list one-by-one in sequence (blocking until it finishes fetching each one).