Replies: 5 comments 1 reply
-
Speaking of "doing something wrong", I think the above example may be accidentally exponential (b/c write amplification, perhaps?) based on the amount of log spam I'm seeing. :) |
Beta Was this translation helpful? Give feedback.
-
Also, just in case it wasn't clear: this really is just a, "hey, this works and it's neat!" issue, not "please support this in addition to your tested, stable clustering model." Feel free to close w/o further discussion, and I can let you know if I go further with my experiments. :) |
Beta Was this translation helpful? Give feedback.
-
Sounds cool! So, how would you describe the workings of the ring? Is it multiple servers with their own local db that distribute changes to all other known peers? |
Beta Was this translation helpful? Give feedback.
-
So consistent hashing maps a key to one of a set of known servers from a "ring", with the expectation that if a node is down you can simply move "around the ring" to find the next appropriate host. Each server ends up with a roughly balanced set of keys to manage, assuming the access patterns are similarly balanced. (There's no idea of "hot" vs. "cold" keys, so if the distribution of reads and writes is highly skewed, one server can end up shouldering a an disproportionate amount of load.) In this implementation, a peer is both a server and a client to the other peers in the ring. Writes are cached locally per the usual I've combined the client and server into one process in this case just to simplify the example, but you could just as easily let clients handle the writes, with each "peer" in the ring being a normal Obviously there's no guarantee of atomic transactions across peers, and writes could easily conflict, but many of those are the same properties that disconnected/offline writes bring to the system already. The model I'd like to prove out is something like git or UUCP, where any given node can arbitrarily write to its own local store, eagerly replicate to the ring when connected, and then re-hash and replicate to new servers as they become available. The server pool is also statically configured in my example, but you can easily add and remove nodes on the ring at runtime, so there's nothing preventing you from doing server discovery via e.g. mDNS, rendezvous announcements, or a distributed hash table. The "rendezvous" model is particularly cool because while you do need a locally-reachable server to discover what peers are available, that server doesn't actually see any of the replicated traffic and so doesn't need be a trusted host. That's also where WebRTC could slot in: using a simple websocket server for peer discovery, then tunnel the actual sync traffic over the WebRTC data channel once you've established a direct connection. Unlike the IPC server, the discovery service does need to be directly accessible to clients, but it can also be entirely stateless and unprivileged. Footnotes
|
Beta Was this translation helpful? Give feedback.
-
Awesome! What you can do to prevent roundtrips of mutations is to add info to the context when replicating to other servers, so that when a mutation sent from server A to B to C comes back around to A, it'll know that was my mutation, I can ignore it - preventing endless loops. Or even better, all servers adding their own id to the context variable so they won't forward to ones in the list already. Really interesting stuff you're working on, keep me updated! |
Beta Was this translation helpful? Give feedback.
-
So this is kind of a wild experiment, but I decided to spend a minute building a proof of concept to replicate data across acebase servers using a hash ring (ala memcached) instead of a single global IPC server:
https://github.com/rcoder/acebase-ring
The bulk of the actual replication logic is here: https://github.com/rcoder/acebase-ring/blob/main/src/server.ts#L80-L195
I'm almost certainly doing something wrong, and the code is just a big mess, but it's an interesting thought experiment at least. The advantage over a single IPC coordinator is of course the lack of a single coordinator that is persistently connected to all of the storage servers. (I.e., it's a true p2p cluster model.)
If I really find myself with time to burn I might try connecting replicas via WebRTC, but that's a science project for another day. :)
Beta Was this translation helpful? Give feedback.
All reactions