standalone or orchestrator?

[shlomi-noach]

The purpose of this service is to provide with an answer "is it OK to write to the database at this time?".

At this time the service is targeted at MySQL clusters, and bases its response on replication lag. E.g. if all replicas are lagging < 800ms, the answer may be "yes, go ahead and write".

To achieve this functionality the service needs to be aware of the replication topologies, or at least of the relevant replicas. We don't need to consult every single replica in a cluster. Not all are created equal.

Orchestrator

This type of information falls conveniently within orchestrator's jurisdiction. orchestrator is familiar with the replication topologies, has knowledge about the pools, can (and does) access replicas to check their lag, provides command line, API & web based interfaces.

This Issue discusses reasons for [not] choosing to implement the functionality of this service directly within orchestrator

Orchestrator: pro

• orchestrator already knows the topologies. If this throttler service is to eb standalone, it would likely talk to orchestrator or similar to get the list of relevant replicas. It would certainly not do discovery.
• orchestrator already configured with credentials, type of query to ask, has the code to query MySQL servers. A standalone service would copy+paste some existing functionality from orchestrator.
• orchestrator already implements a leadership algorithm.
• Meta reason: doing this via orchestrator means more developers getting to know orchestrator.
• Meta reason: existing community users of orchestrator would get this functionality "for free".
• Meta reason: (internal) overhead of yet-another-setup
• Irrelevant meta: if we run this within orchestrator we don't need to come up with a new name

Orchestrator: why not

• orchestrator persists data ; this service does not need to, and persisting would be a painful overhead.
• orchestrator polls servers once every X seconds; this service would poll many times per second
• as result of the above two bullets, running this within orchestrator would mean a change in paradigm, and rather "a different beast" within orchestrator
• the type of communication between nodes of this service is different than the type of communication between orchestrator nodes, at least until orchestrator supports group communication. orchestrator uses (at this time) persistence to share data. Our service would not want to use persistence (at least almost wouldn't want to, I can think of a couple things that would be persisted).
• Half-meta: at some time in the future this service may apply to other datastores than MySQL, and based on other metrics than replicaiton-lag. Binding it to orchestrator would bind it as a MySQL-specific solution.
• Meta: if this goes open-source, many users would benefit from such service. But running orchestrator just for the sake of throttling would be an overkill. I suspect if this were done via orchestrator people would defer from downloading/running this service
• Meta: if we choose to develop this as a standalone service, we can rather easily, later on, integrate it into orchestrator. The reverse is not true. It would be difficult to extract it from orchestrator.

Current thoughts

My gut feeling says to develop this as standalone. And then we'll see.
It's a small enough project that this should ship quickly.

cc @github/platform-data @github/database-infrastructure @skottler @carlosmn

[tomkrouper]

Vote? Add comments below if so desired, otherwise, just vote.

👍 Stand alone
❤️ Orchestrator

[shlomi-noach]

This goes standalone. Thank you for casting your votes!

[shlomi-noach]

Thoughts so far: I'm glad this isn't part of orchestrator. It's just different.