Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Reducing CPU usage for Vitess #4191
As Vitess starts to get used in a variety of platforms, one commonly voiced request has been to reduce its CPU utilization. This is particularly relevant for cloud users for whom CPU is relatively more expensive than other resources.
Additionally, the network in certain environments is not very fast or reliable. This results in elevated tail latencies.
The proposed design is inspired by the existing vtcombo tool, but is different from it. VTCombo is a testing tool that encapsulates all of vitess in a single process. Using a tool like this in place of vtgate will allow it to directly send traffic to mysqls without going through vttablet. This will provide many savings:
Note that the cost of the network hop is mainly relevant if vttablet is also connecting to mysql from an external machine. This is the case when vitess is deployed against already running mysqls, or managed instances like RDS. In cases where vttablet connects to mysql through a socket, only the CPU costs are saved.
However, vtcombo itself will not work in a distributed production setup because it cannot track changing lifecycles of tablets and keyspaces. Also, vtcombo cannot perform other house keeping tasks like backup/restore, resharding, etc.
Instead, we'll preserve most of the existing pieces of vitess as is. This means that vttablets will still remain part of the infrastructure. The only difference, as mentioned above, is that vtgates will send queries directly to mysql.
VTGate has an abstraction layer called Gateway. Underneath this, there is an implementation called DiscoveryGateway that keeps track of vttablets that are going up and down. This layer also contains connections to vttablets.
The connection to vttablet can be replaced by the QueryService of vttablet. It doesn’t even have to be wrapped because the connection API and QueryService API are identical. An example of how to instantiate just the query serving part of vttablet is shown in go/vt/vttablet/endtoend, which tests that part.
This change doesn't come for free. Some features derived from going through a single vttablet will be lost. For example:
The setup changes will also be significant. The connection pools will have to be redistributed across multiple vtgates. The overall connections will have to be increase to accommodate unbalanced traffic.
Some of the monitoring variables that were previously associated to a query service will have to be exported differently at the vtgate level.
VTTablet’s QueryService can remain enabled. This will allow tools like vtworker to send queries to it. It will continue to perform other house-keeping tasks like vreplication and health check reporting.
We can consider moving some of the functionality around in such a way that a vttablet will not be required for normal query serving. That way, we spin them up only when housekeeping work needs to be performed.
We used to have LRU in the past. The problem with that type of scheme is that a spike in traffic causes all connections to be opened at once. Sometimes mysql can't handle it, and sometimes, the container goes OOM, etc. So, we decided to stick to round-robin, which was more steady on resource demands.
But maybe we can reconsider.
Message queue is another feature we'll have to disable (I'll add it to the list).
As for query service disabling, the most critical one is for reparenting and master migrate served types right? In such cases, we'll have to rely on mysql's read-only mode. This is something we are anyway used to in the case of an externally managed mysql.
Can you think of any other workflows where we'd need to disable query service?
On the migrate served types, we also use the query service disabling on the slaves, to prevent using a replica tablet on the wrong shard / keyspace. Right now we ensure no-one reads from the wrong slave tablet by disabling the query service, or adding table black lists. This is less critical, as it's more to keep the vtgates in sync with the tablets right now, and you'll have that covered.
Id generation: more gaps, and no real increasing IDs. We used to guarantee increasing IDs, with holes. Now we can just guarantee unicity.
You can track progress of this implementation here: https://github.com/sougou/vitess/tree/vtdirect.
The embedded TabletServer additionally encapsulates a connection to vttablet and proxies its health while also mimicking the corresponding state changes.
As for IDs, I'm thinking of (identifying and) forwarding those requests to the vttablet instead. This approach should work for messages also.