partial full connectivity upload/download

[Eknir]

Epic

Story

As a Swarm node which is only connected to one other Swarm node, I want that my chunk is stored at the node whose address is closest to the address of the chunk, such that anybody who requests the chunk afterwards, knows where to find this chunk.

Background

This epic builds on top of #48 and #49, but adds the forwarding of push-sync and retrieval messages. If a node is only connected to one other Swarm node, this one Swarm node always must forward the push-sync/retrieval messages.

Acceptance criteria

• Given a node who is only connected to one (random) other node, an upload from this particular node reaches the destination (closest node).
• Given a chunk that exists in the network, the node who is only connected to one (random) other node can download this chunk from the network

[acud]

There's a few problems with this ticket:

• a node that connects to another node, as of now, will never remain connected to that one node, because it implicitly participates in discovery, meaning it will receive information about new peers immediately upon connecting to one node. that other node's peers will also receive information about this node and will potentially try to connect to it upon receiving the information
• another unclear point is if the loner node that wants to connect to one other node and that's it is supposed to store chunks at all (so, that the edge case of loner's address is closest to chunk address is satisfied)
• it seems like this ticket attempts to adapt somehow the concept of light nodes from the old codebase, but this doesn't hold with the capabilities of the new one. it won't hold with full connectivity that we have right now. with kademlia potentially yes but still light node notion would have to be introduced

[Eknir]

A node that connects to another node, as of now, will never remain connected to that one node, because it implicitly participates in discovery, meaning it will receive information about new peers immediately upon connecting to one node. that other node's peers will also receive information about this node and will potentially try to connect to it upon receiving the information.

This, then, must be possible to switch off

another unclear point is if the loner node that wants to connect to one other node and that's it is supposed to store chunks at all (so, that the edge case of loner's address is closest to chunk address is satisfied)

If the loner node's address is closest to the chunk, he stores the chunk

it seems like this ticket attempts to adapt somehow the concept of light nodes from the old codebase, but this doesn't hold with the capabilities of the new one. it won't hold with full connectivity that we have right now. with kademlia potentially yes but still light node notion would have to be introduced

Can you clarify this?

All of the above answers (and the ticket itself) assume that it is trivial to make a node to be only connected to one other node. If this is not the case, perhaps, the ability to do forwarding is best satisfied after we have implemented Kademlia. Certainly, it is not worth adding additional complexity to the code, just for allowing a node to have just one connection. If, however, this feature is useful (perhaps for testing or something else), it might be worth putting this effort. Tagging @janos , as he perhaps has a better idea on the technical implementation of this issue.

[acud]

If the loner node's address is closest to the chunk, he stores the chunk

does not compute. at least, not very clearly. if it is within this node's responsibility to store something on the network level, then it infers this node participates in the topology, which also means this node should try and connect to as much nodes as it can, because if it consents to assuming the responsibilities of being a member of an arbitrary nearest neighbourhood, then it means it should also bear the responsibility of connecting to other nodes in that neighbourhood too (and make itself available for the other nodes in the neighbourhood to connect with)

[acud]

the ability to do forwarding is best satisfied after we have implemented Kademlia

The ability to do forwarding will be implemented in the current iteration of push sync. What you are describing here is not forwarding per se, because forwarding means i will forward a chunk to a peer closer to that chunk than me. This ticket is orthogonal to classic notion of forwarding by that that it asks a node to forward everything we send it with, according to its address book. This is not classical forwarding due to particular permutations of the (let's get to it then...) kademlia table of the node which is "forwarding" to your definition, that is, it might be that this node will have to forward chunks to nodes farther away from it under some circumstances.

[Eknir]

I suggest we have a conversation with @janos , as he agreed with this definition of the user story. I see you are raising technical issues with the actual implementation, so let's wait and see what he thinks about it.

If it indeed is not wise to implement now, I see two possibilities

• We defer implementation of forwarding to after we have the ability to build a Kademlia Topology
• We implement it now, but only validate workings with unit tests. We wait until after we have Kademlia Topology to validate whether it is actually working with integration tests.

Do you have another suggestion?

[janos]

Sorry for late response. I think that this epic is only about forwarding to the closest node and that more complex forwarding can be added explicitly with other tasks to address edge cases and more complex algorithms, kademlia forwarding with other parameters as price based forwarding and maybe some other types if we have more ideas.