Error: pin: context deadline exceeded

[davidar]

Pinning seems to time out on large (10GB) directories:

# load into cache
$ ipfs refs -r QmYaFMPTgDkwp3vun6o3zfVv6gFwDWVSJWwZQHieXj1Ys6
<snip>
# make sure it's definitely cached
$ ipfs get QmYaFMPTgDkwp3vun6o3zfVv6gFwDWVSJWwZQHieXj1Ys6

$ ipfs pin add -r QmYaFMPTgDkwp3vun6o3zfVv6gFwDWVSJWwZQHieXj1Ys6
Error: pin: context deadline exceeded

Affects: ipfs-inactive/archives#2

Edit: Actually, it would also be nice if you didn't have to do the caching step separately (i.e. make pin act like get, but without actually creating files in the current directory).

[whyrusleeping]

@davidar thats interesting. how long did the pin call take?

[davidar]

@whyrusleeping A minute or two before error, I guess (refs -r also takes a while just to list blocks from the cache). I can time it exactly later today if that would help.

[rht]

Actually, it would also be nice if you didn't have to do the caching step separately (i.e. make pin act like get, but without actually creating files in the current directory).

Related #875

[whyrusleeping]

I've got to be honest, i cant tell where that timeout in pin is coming from... @rht can you see where?

[davidar]

@rht not exactly, what I mean is download to the block store and pin it simultaneously (without timing out). A progress bar too would be nice

[rht]

@whyrusleeping In corerepo.Pin, it comes from either

1. core.Resolve(ctx,...) which calls n.Resolver(ctx,...) (which timeout happens only in path.Resolver.ResolveLinks)
2. n.Pinning.Pin(ctx, ...) which calls Blockservice.GetBlock (which timeout happens somewhere in bitswap).

Currently the two can't be distinguished because both has error with prefix "pin: ".
Maybe @davidar can change the message in https://github.com/ipfs/go-ipfs/blob/master/core/corerepo/pinning.go#L32 to "pin: resolve: ", rebuild, then rerun the ipfs pin add and see whether the error message has this prefix?

Edit: clarified the path

[rht]

@davidar I ref-ed the issue because this is another view of not wanting to "duplicate" files (you only want the cache).

[rht]

Actually, since ipfs refs -r uses core.Resolve and it works just fine, the err is constrained to come from the 2nd one (n.Pinning.Pin). @davidar sorry no need to run the test.

[whyrusleeping]

@rht it has to be the one in ResolveLinks, i'm fairly certain that bitswap wont ever timeout a request.

[rht]

ipfs refs uses https://github.com/ipfs/go-ipfs/blob/master/core/commands/refs.go#L192-L201 which uses ResolveLinks, which is identical to the first part of pin https://github.com/ipfs/go-ipfs/blob/master/core/corerepo/pinning.go#L28-L35

[rht]

Wish this could return an iterator rht@6d55bc9

[jbenet]

@rht can use channels as iterators

i like iterators in python. probably the best support i've seen in a language for them. (without going c++ operator overloading crazy).

[whyrusleeping]

This has since been improved, ipfs pin add is much better about timeouts and resource consumption