Detect and signal schema changes on vttablets

[systay]

Detect schema changes on the tablet.

With this PR, a vttablet that finds itself as a master, will periodically check if schema has changed on the underlying mysqld. If any schema changes are found, the changed table names will be sent to any listening vtgates using the pre existing health check mechanism.

If this feature is enabled, a database will be created on the underlying MySQL instance named _vt, where meta-information about the schema will be stored.

Related to #7994

[shlomi-noach]

If this feature is enabled, a database will be created on the underlying MySQL instance named _vt

I think _vt is always created, if only for the heartbeat mechanism.

[shlomi-noach]

Elaborating on my previous comment: _vt schema is always created, within a few seconds of tablet startup. There's multiple parties that would create it, but onlineddl/Executor is one I'm most familiar with.

Creation:

vitess/go/vt/vttablet/onlineddl/executor.go

Line 2292 in 0ce71c7

if err := e.initSchema(ctx); err != nil {

Called by tablet state manager, Open(), right after tablet is ready.

Which runs these queries:

vitess/go/vt/vttablet/onlineddl/schema.go

Lines 345 to 359 in 0ce71c7

	var applyDDL = []string{
	sqlCreateSidecarDB,
	sqlCreateSchemaMigrationsTable,
	alterSchemaMigrationsTableRetries,
	alterSchemaMigrationsTableTablet,
	alterSchemaMigrationsTableArtifacts,
	alterSchemaMigrationsTableTabletFailure,
	alterSchemaMigrationsTableTabletFailureIndex,
	alterSchemaMigrationsTableProgress,
	alterSchemaMigrationsTableContext,
	alterSchemaMigrationsTableDDLAction,
	alterSchemaMigrationsTableMessage,
	alterSchemaMigrationsTableTableCompleteIndex,
	alterSchemaMigrationsTableETASeconds,
	}

Which, in particular, include:

vitess/go/vt/vttablet/onlineddl/schema.go

Line 21 in 0ce71c7

sqlCreateSidecarDB = "create database if not exists _vt"

[shlomi-noach]

one bug found.

Fear mongering alert: I guess my reservation is that every X minutes (30 in this code) we're going to query information_schema for all tables in a schema, which can be tens of thousands of tables in some environments. I actually never tried querying I_S for that many tables, and I fear this might be a risky operation.

[shlomi-noach]

I guess this code already exists. But it's unclear from the name queryserver-config-schema-reload-time what the units are. I'd assume it was a Duration, but it's in fact seconds. I guess too later to change the name,

[systay]

Fear mongering alert: I guess my reservation is that every X minutes (30 in this code) we're going to query information_schema for all tables in a schema, which can be tens of thousands of tables in some environments. I actually never tried querying I_S for that many tables, and I fear this might be a risky operation.

This one of the reasons we are adding a flag for this feature. We don't think that it will be something everyone uses. The use case you mention sounds like a situation where you might not want to keep track of schema like this.

[shlomi-noach]

Just curious why 2.1 😄

[harshit-gangal]

2 seconds check related to no schema change was added by @vmg to avoid query cache test to fail. To make the test faster made it to 2.1, i wanted to make it 2.0000001 but that would be too ambitious

[shlomi-noach]

Then I think 2.1 might also be too ambitious. Might introduce flakyness?