Autodiscovery - autosetup of cluster members

[hsanjuan]

Given a single existing cluster member, a new cluster node should be able to set itself up, retrieve and connect to all members of the cluster.

Note the trickiness of this:

• At least some components are not ready to work or should not attempt to work (consensus) before receiving the list of peers
• It's not as easy as connecting to someone and retrieving current cluster members (i.e. would fail when starting a bunch of new members at the same time). Also involves broadcasting that a new node is available.
• Need to give more thought. Should aim for something simple and straightfoward. Errors during automated setup are the worst.

[meyerzinn]

Is it necessary for every peer to be aware of another in ipfs-cluster? I was imagining something like a private DHT that assigns blocks to nodes.

[hsanjuan]

Cluster shares state (list of pinned things) using a consensus algorithm (Raft) so all nodes need to know each other.

[meyerzinn]

@hsanjuan When starting a bunch at once, why not seed with one node like so:

Here's a system I was thinking about for autodiscovery. Suppose we have three nodes: A, B, and C. Discovery takes place as follows:

Step 1: Seeding

In order for the system to function, all nodes must be "seeded" with at least one other node. In this diagram, node A is joining the cluster by contacting C.

                            +--------+
                            |        |
                            | Node B |
                            |        |
+--------+                  +--------+
|        |
| Node A |
|        |
+----+---+
     |
     |
     |                        +--------+
     |                        |        |
     +------------------------> Node C |
                              |        |
                              +--------+

Step 2: Pass-through

In this step, peers wishing to join the cluster contact a current member of the cluster. In this diagram, B wishes to join the cluster; thus, B contacts C.

                            +--------+
                            |        |
                            | Node B |
                            |        |
+--------+                  +----+---+
|        |                       |
| Node A |                       |
|        |                       |
+----+---+                       |
     |                           |
     |                           |
     |                        +--v-----+
     |                        |        |
     +------------------------> Node C |
                              |        |
                              +--------+

Step 3: Repeat

C returns a list of peers it is immediately connected to, and B goes to each of those peers individually and requests a list of peers. In this diagram, B contacts A, but since the only peer A will return is C, and B is already connected to C, this "trail" has gone cold.

                            +--------+
                            |        |
     +----------------------+ Node B |
     |                      |        |
+----v---+                  +----+---+
|        |                       |
| Node A |                       |
|        |                       |
+----+---+                       |
     |                           |
     |                           |
     |                        +--v-----+
     |                        |        |
     +------------------------> Node C |
                              |        |
                              +--------+

This mechanism works really well when considering a more complicated model, like so:

                       +--------+
                       |        |
                       | Node A |
                       |        |
                       +---+----+
                           |
+--------+             +---v----+           +--------+
|        |             |        |           |        |
| Node D +-------------> Node B |           | Node F |
|        |             |        |           |        |
+---^----+             +----^---+           +---+----+
    |                       |                   |
    |                  +----+---+               |
    |                  |        |               |
    |                  | Node C <---------------+
    |                  |        |
+---+----+             +--------+
|        |
| Node E |
|        |
+--------+

Let's assume that nodes are seeded with Node B. No matter what order nodes connect in, they should all end up in the same cluster.

[hsanjuan]

@20zinnm thanks for the comments. For the moment I'm not supporting the case where everyone bootstraps from someone else at the same time.

C returns a list of peers it is immediately connected to, and B goes to each of those peers individually and requests a list of peers

This is a graph search, but in cluster C should know all its nodes and if it doesn't yet it should tell B and everyone else about any modifications (trails may warm up). Since it's not just about having a graph, but about having everyone connected to everyone else, this grows very complex.

An easier solution is to have nodes joining running clusters where they can receive the full list of peers from whoever they connect to directly, and where it is possibly to push the new update to all peers. After joining the full peer list is saved to the configuration for future use on start up. This also re-uses most of the PeerAdd() code path, as B is basically requesting the bootstrap node to PeerAdd(B).

[meyerzinn]

@hsanjuan Why would it need to announce modifications if the seeker needs to contact all nodes anyways? In order to establish connections, it needs to contact the nodes anyways, so it could announce its own "joining."

Also, the running cluster means that the ability of a node to join is dependant on (a) how up-to-date the cluster is, and (b) whether or not there is already a cluster. You also make it impossible to create a new cluster automatically without pre-filling peer lists, so my solution is the same as yours in the abstract except the peers are discovered at runtime.

[hsanjuan]

ok @20zinnm I'm gonna give a try to the way that you propose it. It is not way more complicated and the result is better.

[meyerzinn]

The only difference in practice is that mine only mandates the seeding of one node, which could be done by a Kubernetes service or whatnot. It requires no persistence and only one entry point. If you would like, I could put together a distributed demo cluster (obviously not IPFS, just a simple demo cluster).

[hsanjuan]

@20zinnm I know, my original solution would also work with 1 seeding node which everyone uses. The new one would theoretically work even when using several seeding nodes and everyone connects at the same time. For this the "newcomer checks-in with every node" way that you propose is mentally simpler to me than adapting my original approach.

I have a more or less working solution but I wanna give it another round.

[hsanjuan]

@20zinnm went a third way in the end by commiting new members to the distributed log rather than broadcasting and recursively contacting peers. I have realized it is very easy to mess the consensus when someone does not cooperate perfectly or peers have disjoint logs, or different leadership terms etc. Despite locking sections of the code I wasn't very happy with the results and added complexity.

As of the latest approach, a cluster should be able to bootstrap from the same node (at the same time).

[meyerzinn]

@hsanjuan What do you mean by distributed log? What implementation is the log using that wouldn't suffer the same faults?

I think it's fine to have everyone join through the same node but it's not scalable in my opinion. My check-in method means you can bootstrap a thousand nodes by giving each a pointer to the one before it, without having to spam a single member for a list.