swarm/network/stream: added pure retrieval test (syncing disabled)

[holisticode]

his PR adds a pure retrieval test to the snapshot retrieval tests.

Up to today, the retrieval tests were uploading chunks randomly to nodes and then trying to download those chunks from every node. To upload chunks, the FileStore.Store() function was being used - essentially uploading chunks to individual nodes directly. For retrieval to work in such a scenario, we need to have syncing enabled in order for chunks to be stored at their actual expected nodes.

This new test bypasses syncing by evaluating which chunk would be expected to be at which node, then storing chunks directly in those nodes's LocalStore - essentially simulating syncing by accessing stores directly (simulation framework allows that). This way, the test simulation run can switch off syncing and directly starts retrieving chunks, as if syncing would have already terminated.

[zelig]

why not put this part outside the simulation Run just as part of setup?

[holisticode]

Strictly speaking, if we want the UpNodeIDs() (the nodes which are Up), then we should query this information when the simulation is started, thus inside sim.Run()

[zelig]

so this test suits the old action, trigger, check pattern

[zelig]

that would also make it concurrent, which is also better for testing realistic scenarios.
I would just allocate chunks to nodes to retrieve them independently, otherwise caching may distort the pure nature of this test

[holisticode]

I am not sure I understand what you mean here and I am also not sure what action I should take on your comments. I have thus currently not changed anything

[zelig]

https://github.com/ethersphere/go-ethereum/tree/master/p2p/simulations#testing-framework

an example is https://github.com/ethersphere/go-ethereum/blob/master/swarm/network/simulations/discovery/discovery_test.go#L221

[zelig]

if we retrieve each chunk from each node, there is a lot of caching happening, so it may shield retrieval issues

[holisticode]

In some way I agree but I finally disagree.

Retrieval in Swarm is intrinsically linked to caching. This is something you yourself have been raising all the time, when saying that we actually can't switch off caching.

So my conclusion is that this in turn is in fact also part of retrieval, so we should leave it as-is. I would furthermore not even know how to circumvent this situation, and finally, you should be able to specify what you mean by "retrieval issues" in order to do something about it.

[zelig]

do we need this part?
or why check size only?

[zelig]

"storing chunk at the peer whose overlay address is closest to chunk hash`