Why is the config distributed from a central point

[tionis]

Hi, interesting project you build here.
The main question I have is why a central s3 config distribution point is needed. This introduces more infrastructure to host or buy and is a central point of failure. As far as I understand the design distributing the config via gossip or some other direct p2p per-node way would be better distributed.
Is there a reason the central s3-based distribution was chosen, beyond just ease of use and implementation?

[encodeous]

Hey, that's a good point.

I mainly chose a central distribution channel for the sake of simplicity and reliability. It's much easier to ensure the whole network is consistent this way.

With some kind of p2p distribution, we need to make sure the whole network gets the updated configuration before applying it. Otherwise you can imagine a case where we "lose" some nodes by entering a faulty state. E.g: config is partially applied, so a node is not reachable by other nodes, therefore the node can't receive the config update...

I'm sure there might be some neat way to do this, I just thought that could be addressed later haha.

Could be something worth looking into if you have the time.

[tionis]

I see, that makes sense. I didn't think about the issue of the config not being applied at the same time. But the current implementation also just applies it once fetched, so perhaps updates distributed via p2p could have an embedded apply timestamp to allow for it to be distributed through the network. This of course does not guarantee anything and does not scale well, but would remove a central point.