Move MySQL schema collection to the shared SchemaCollector and remove the legacy collector

[eric-weaver]

What does this PR do?

Reworks MySQL schema collection onto the shared SchemaCollector base class so every
collection path streams schema metadata in chunks and emits the collection_payloads_count
snapshot markers the backend uses to detect complete point-in-time snapshots.

• Adds MySqlSchemaCollector (datadog_checks/mysql/schemas.py) with two strategies that
  produce byte-identical payloads:
  • single_query: one JSON-aggregation query per database, streamed one row (table) at a
    time via an unbuffered server-side cursor.
  • chunked: streams the table list and fetches column / index / foreign-key / partition
    detail per chunk with flat INFORMATION_SCHEMA queries.
• Selects the strategy automatically by server version: single_query on JSON-capable
  servers (MySQL >= 5.7.22 / MariaDB >= 10.5.0) and chunked on older versions (works on
  every supported version). A hidden collection_strategy option can force either strategy
  for debugging.
• Removes the legacy DatabasesData collector, which used buffered cursors and could not emit
  snapshot markers, so keeping it would leave older servers without consistent snapshots.

Behavior change to note: the legacy collector, on exceeding max_execution_time, emitted a
partial payload plus a collection_errors: [{error_type: "truncated"}] marker. The shared
collector enforces max_execution_time at the SQL level; if a database's query exceeds it,
that database is skipped (already-flushed chunk payloads are retained) and no truncated
marker is emitted.

Motivation

The legacy collector issues 1 + 4 x ceil(tables/500) round trips per database and buffers
results, so both round trips and client memory scale with table count. The single_query
path streams unbuffered and issues ~1 query per database.

Benchmarks below drive the real collector code against a live server via a minimal fake check,
capturing payloads in-process (synthetic schema: 12 columns/table, a PK, two secondary indexes,
a foreign key to a sibling table, range partitions on every 10th table).

Scaling table count (MySQL 8.0.46, localhost)

peak mem is the Python tracemalloc peak during collection (emitted payloads discarded so the
figure reflects the collector, not the harness).

tables	dbs	mode	time (s)	payloads	bytes	queries	peak mem (KiB)
600	3	legacy	0.131	3	1,637,373	16	n/a
600	3	single_query	0.152	3	1,557,978	4	n/a
600	3	chunked	0.127	3	1,557,978	16	n/a
5,000	5	legacy	4.11	5	13,636,157	46	20,522
5,000	5	single_query	3.21	15	12,979,395	6	12,039
5,000	5	chunked	3.28	15	12,979,395	46	17,754
20,000	10	legacy	22.45	10	54,540,316	171	39,872
20,000	10	single_query	18.52	50	51,914,110	11	12,041
20,000	10	chunked	19.73	50	51,914,110	171	18,328

• Memory: single_query stays flat at ~12 MiB from 5k to 20k tables (row-at-a-time
  streaming, per-table aggregation done server-side). Legacy nearly doubles (20.5 -> 39.9 MiB);
  chunked sits in between (~18 MiB).
• Round trips: single_query issues ~1 query per database (11 for 10 dbs); legacy and
  chunked scale with table count (171 at 20k tables).
• Time: single_query is fastest even on localhost (~17-22% faster than legacy), and the
  gap grows with network latency (below).
• Correctness: single_query and chunked are byte-identical at every scale.

Database-side impact (5,000 tables, 5 dbs, MySQL 8.0.46)

Measured via session-scoped SHOW SESSION STATUS deltas around each run.

mode	server bytes_sent	tmp_tables	tmp_disk_tables	rows_read
legacy	5,406,533	30	0	3,662,713
single_query	15,212,448	40	0	3,673,182
chunked	5,406,533	30	0	3,662,375

• Server read work is equal (~3.66M handler rows for all three): single_query does not
  make the server scan more of information_schema.
• No disk spill: the extra temp tables from JSON grouping stay in memory
  (tmp_disk_tables = 0) at this scale.
• Tradeoff: single_query sends ~2.8x more bytes because server-side JSON repeats object
  keys per row; the client normalizes this back down to a similar final payload. So it trades
  higher DB egress for fewer round trips, bounded client memory, and equal server reads.

Network-latency sweep (MySQL 8.0, 5,000 tables, Toxiproxy on server->client)

added latency	single_query	chunked	legacy
0 ms	3.37 s	3.48 s	4.33 s
1 ms	3.38 s	3.52 s	4.31 s
5 ms	3.41 s	3.74 s	4.44 s
25 ms	3.43 s	4.96 s	5.54 s

single_query (6 round trips) changes little across the range (+0.06 s from 0 to 25 ms), while
chunked and legacy (46 round trips each) grow by ~1.2-1.5 s over the same range, reflecting
the lower round-trip count on a latency-bound link.

Cross-version payload equivalence

single_query output was deep-compared against the legacy collector on every supported version
(reassembling chunked payloads into per-database table maps; envelope fields ignored):

version	result
MySQL 5.7.44	MATCH
MySQL 8.0.46	MATCH (also verified at 20,000 tables)
MySQL 8.4.10	MATCH
MariaDB 10.5.29	MATCH
MariaDB 10.11.18	MATCH
MariaDB 11.4.12	MATCH

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[datadog-datadog-prod-us1]

 

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 95.17%
• Overall Coverage: 90.49% (+2.12%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: fe08b50 | Docs | Datadog PR Page | Give us feedback!}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7eeeb9d8ce

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Preserve max_execution_time for chunked collection

When the automatic fallback selects chunked on older MySQL/MariaDB versions, or when collection_strategy: chunked is forced, this branch returns before applying any SQL timeout/deadline; _iter_chunked_tables() then runs the table query plus all detail queries without checking self._config.max_execution_time. That means large or slow schemas can run far beyond the documented collect_schemas.max_execution_time limit, unlike the single-query path and the legacy collector's truncation behavior.

Useful? React with 👍 / 👎.

[dd-octo-sts]

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
agent-reqs	Verify check versions match the Agent requirements file	✅
ci	Validate CI configuration and code coverage settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
config	Validate default configuration files against spec.yaml	✅
dep	Verify dependency pins are consistent and Agent-compatible	✅
http	Validate integrations use the HTTP wrapper correctly	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
labeler	Validate PR labeler config matches integration directories	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
license-headers	Validate Python files have proper license headers	✅
licenses	Validate third-party license attribution list	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
openmetrics	Validate OpenMetrics integrations disable the metric limit	✅
package	Validate Python package metadata and naming	✅
qa-label	Validate the pull request declares whether it needs QA for the next Agent release	✅
readmes	Validate README files have required sections	✅
saved-views	Validate saved view JSON file structure and fields	✅
version	Validate version consistency between package and changelog	✅

View full run