localstore: fix index leak when NN chunks enter localstore

[acud]

This PR fixes an index leak that was manifesting when chunks that would enter the node's local storage would, under certain connectivity conditions be not added to the GC index.

This is due to the fact that the uploader node would be the closest node to the chunk, thus again, under certain connectivity condition this would result in the chunk never syncing, and the retrievalDataIndex eventually growing out of bound, since no matter how many chunks in lower bins would be garbage collected, the higher bins would always make the store size eventually overflow.

This fix introduces a pluggable function to localstore that is called when a chunk enters the local storage.

During implementation several weaknesses were unraveled:

1. a node with very little storage could GC the content uploaded before it is pull synced
2. as a result, anonymous content syncing on light nodes could be severely damaged
3. a flag indicating that the node is actively waiting for an upload to sync should probably be added to Swarm in order to short-circuit this mechanism for ephemeral nodes

fixes #1920

[janos]

The code cannot be compiled:

storage/localstore/mode_put_test.go:325:19: cannot use func literal (type func(chunk.Address) bool) as type func([]byte) bool in field value

[janos]

LGTM. 👍

[nolash]

I see @zelig reviewed it while I had breakfast halfway. Anyway, curious about the negative bin id

Also curious about the description;

as a result, anonymous content syncing on light nodes could be severely damaged

what is anonmous content syncing on light nodes?!

under certain connectivity condition

I would like descriptions to be more precise, helps us puny humans that don't have context to understand better and do better review work.

[acud]

@nolash the issue is as follows:
In the old syncer an upstream peer would tell the downstream peer on which bins to ask for chunks on (subscriptions). this would guarantee, for example, that the upstream peer would establish subscriptions on bins within depth, although that upstream peer might have the downstream peer outside of it's saturation depth. this would also always guarantee that all chunks will get synced from the downstream peer at some point or another (since subscriptions were established according to the downstream peer's perspective).

in the new stream this is not the case. upstream peers establish streams according to their own kademlia and regardless of the downstream peer's connectivity. while this is more resilient implementation (and less attack prone), it did not in its naive implementation cater for the fact that the downstream peer might be the closest node to certain chunks. the connectivity pattern that was manifesting was that the downstream peer appeared outside of saturation depth of other nodes, thus syncing of certain chunks was not bound to happen (since the downstream peer was the closest one and the upstream peers all had not been subscribed to that node's higher bins since it was outside of their depth). this therefore manifested in a localstore leak since some chunks were never Set therefore never added to gcIndex therefore never deleted.

[nolash]

@janos I think I understand more or less. Thanks.

[nolash]

Still not sure about the "what is anonmous content syncing on light nodes" though :)