Excessive resource usage up to crash during pinning

[ghost]

Version/Platform/Processor information (from ipfs version --all):

Basically 0.4.3-rc1

> ipfs version --all
20:37:44.910  INFO   cmd/ipfs: IPFS_PATH /data/ipfs main.go:297
go-ipfs version: 0.4.3-dev-566c08e
Repo version: 4
System version: amd64/linux
Golang version: go1.6.3

Type (bug, feature, meta, test failure, question): bug
Area (api, commands, daemon, fuse, etc): bitswap
Priority (from P0: functioning, to P4: operations on fire): 2

Description:

I tried syncing the pins on the gateways, and one particular object keeps reproducibly crashing uranus.i.ipfs.io. Within a minute or two it goes up to 200k goroutines and eventually fails to allocate memory.

> ipfs pin add QmRn43NNNBEibc6m7zVNcS6UusB1u3qTTfyoLmkugbeeGJ

[whyrusleeping]

great, disk contention on the writing provider entries to disk. I'll see if i cant find a clever way to batch these

[ghost]

I'm not sure this is a side effect of writing providers to disk. Yesterday I tried master with the write-providers-to-disk commit reverted (solarnet/v0.4.3-downgrade branch), and it still had this issue. I also tried the latest master after the merge of #2993, and it also showed the same excessive resource usage.

I'm not sure this is a regression in v0.4.3-rc1, but rather a general problem with big object graphs, whatever big means in the case of the hash I mentioned in the opening comment. Should we bump it off rc2?

[whyrusleeping]

yeah, lets bump it off rc2

[ghost]

Resource usage is bounded now: