Skip to content

[mysql] Name REFERENCES in the schema collection privileges warning - #24737

Closed
azhou-datadog wants to merge 12 commits into
masterfrom
allen.zhou/mysql-schema-collection-improvements
Closed

[mysql] Name REFERENCES in the schema collection privileges warning#24737
azhou-datadog wants to merge 12 commits into
masterfrom
allen.zhou/mysql-schema-collection-improvements

Conversation

@azhou-datadog

@azhou-datadog azhou-datadog commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Points the schema-collection insufficient-privileges warning at the REFERENCES privilege instead of SELECT.

MySQL exposes a table in INFORMATION_SCHEMA only to users that hold some privilege on it — that is why schema collection currently asks customers to GRANT SELECT ON *.*, and why the setup docs have to explain in prose that the Agent doesn't actually read their data. REFERENCES satisfies the same visibility check and grants no ability to read (or write) anything.

No query changes: the collected payload is identical under either privilege.

Motivation

Requiring SELECT on customer tables is the main setup hurdle for MySQL schema collection and blocks enabling it by default. A spike verified that REFERENCES produces a byte-identical schema payload while leaving the Agent unable to read table data:

  • All six schema queries diff clean against a full-SELECT baseline on MySQL 5.6.51, 5.7.44, 8.0.45, 8.4.3, 9.7.2, Percona 8.4.10 and MariaDB 10.11.18, 11.4.12.
  • End to end, the mysql_databases payload the Agent puts on the wire is unchanged after swapping the grant (18 tables / 97 columns / 41 indexes / 4 FKs / 5 partitions, empty diff), while SELECT * FROM <table> as the Agent user returns ERROR 1142.
  • REFERENCES alone cannot create a foreign key: that needs CREATE/ALTER on the referencing table, which the datadog user does not have.
  • Grantable on RDS/Aurora (rds_superuser_role includes it WITH GRANT OPTION), Cloud SQL, and Azure MySQL.

Worth noting: tests/conftest.py:606 already grants REFERENCES to the test user, with the comment "needed to query INFORMATION_SCHEMA.REFERENTIAL_CONSTRAINTS in mariadb 10.5 and above". That requirement was worked around in the fixture but never surfaced to customers, so MariaDB users following the docs today collect foreign keys without their ON UPDATE/ON DELETE actions.

One behavioural consequence for the docs rather than the code: with REFERENCES instead of SELECT, the STATEMENT explain strategy (a direct EXPLAIN as the datadog user) no longer works, so explain plans depend on the explain_statement definer procedures that setup already prescribes. Verified both halves in a sandbox.

Additional Notes

Docs counterpart: DataDog/documentation#38606.

Split out of this PR, to be handled separately: SQL_FOREIGN_KEYS aggregates referenced_column_names with group_concat and no ORDER BY, unlike the column_names aggregate beside it, so composite foreign keys report their referenced columns in a non-deterministic order (reproduced as a,b vs b,a on the same constraint across MySQL 5.6 and MariaDB). test_collect_schemas currently sorts that field before comparing, which hides it.

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
  • If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

…posite FK columns

MySQL only exposes a table in INFORMATION_SCHEMA to users holding a privilege on
it, which is why schema collection currently asks customers for SELECT on their
tables. The REFERENCES privilege satisfies that same visibility check without
granting any ability to read table data, so the insufficient-privileges warning
now points at it instead of SELECT.

Also make composite foreign keys deterministic: the referenced_column_names
aggregate had no ORDER BY, unlike the column_names aggregate beside it, so a
multi-column foreign key could report its referenced columns in either order
from one collection to the next. The test previously sorted that field before
comparing, which hid the non-determinism; it now asserts ordinal order directly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@dd-octo-sts

dd-octo-sts Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Validation Report

Validation Description Status
qa-label Validate the pull request declares whether it needs QA for the next Agent release

Run ddev validate all changed --fix to attempt to auto-fix supported validations.

Passed validations (20)
Validation Description Status
agent-reqs Verify check versions match the Agent requirements file
ci Validate CI configuration and code coverage settings
codeowners Validate every integration has a CODEOWNERS entry
config Validate default configuration files against spec.yaml
dep Verify dependency pins are consistent and Agent-compatible
http Validate integrations use the HTTP wrapper correctly
imports Validate check imports do not use deprecated modules
integration-style Validate check code style conventions
jmx-metrics Validate JMX metrics definition files and config
labeler Validate PR labeler config matches integration directories
legacy-signature Validate no integration uses the legacy Agent check signature
license-headers Validate Python files have proper license headers
licenses Validate third-party license attribution list
metadata Validate metadata.csv metric definitions
models Validate configuration data models match spec.yaml
openmetrics Validate OpenMetrics integrations disable the metric limit
package Validate Python package metadata and naming
readmes Validate README files have required sections
saved-views Validate saved view JSON file structure and fields
version Validate version consistency between package and changelog

View full run

@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Jul 30, 2026

Copy link
Copy Markdown

Pipelines  Tests

⚠️ Warnings

🚦 45 Pipeline jobs failed

Check PR | run / Check PR changelog   View in Datadog   GitHub Actions

🔧 Fix in code. This looks caused by changes in this PR. Package "sqlserver" has changes that require a changelog. Please run `ddev release changelog new` to add it.

PR | test / test (linux, ubuntu-22.04, sqlserver, SQL Server on Linux (py3.13-linux-2022-newyork), py3.13-lin... / SQL Server on Linux (py3.13-linux-2022-newyork)-py3.13-linux-2022-newyork   View in Datadog   GitHub Actions

🔧 Fix in code. This looks caused by changes in this PR. Linting failed with 3 errors: import block is unsorted, redefinition of unused `extract_int_value`, and redefinition of unused `extract_value`.

PR | test / test (linux, ubuntu-22.04, sqlserver, SQL Server on Linux (py3.13-linux-2022-tokyo), py3.13-linux... / SQL Server on Linux (py3.13-linux-2022-tokyo)-py3.13-linux-2022-tokyo   View in Datadog   GitHub Actions

🔧 Fix in code. This looks caused by changes in this PR. Linting failed due to unsorted/unformatted imports and redefinitions in datadog_checks/sqlserver/xe_collection/base.py:1.

View all 45 failed jobs.

📋 Copy prompt for your agent
CI on my pull request is failing. Help me find and fix the root cause of each failing job below — they were flagged as caused by changes in this PR, so focus on the diff. For each job, explain the failure and propose a fix.

Branch: allen.zhou/mysql-schema-collection-improvements

Check PR | run / Check PR changelog
Commit: 89e9dd6e5923d7c5f61de4c37a9cb09cd092bdb6
Error (code / build):
Package "sqlserver" has changes that require a changelog. Please run `ddev release changelog new` to add it.
CI job: https://github.com/DataDog/integrations-core/actions/runs/30565361489/job/90948412045

PR | test / test (linux, ubuntu-22.04, sqlserver, SQL Server on Linux (py3.13-linux-2022-newyork), py3.13-lin... / SQL Server on Linux (py3.13-linux-2022-newyork)-py3.13-linux-2022-newyork
Commit: 89e9dd6e5923d7c5f61de4c37a9cb09cd092bdb6
Error (code / quality):
Linting failed with 3 errors: import block is unsorted, redefinition of unused `extract_int_value`, and redefinition of unused `extract_value`.
CI job: https://github.com/DataDog/integrations-core/actions/runs/30565361002/job/90948593122

PR | test / test (linux, ubuntu-22.04, sqlserver, SQL Server on Linux (py3.13-linux-2022-tokyo), py3.13-linux... / SQL Server on Linux (py3.13-linux-2022-tokyo)-py3.13-linux-2022-tokyo
Commit: 89e9dd6e5923d7c5f61de4c37a9cb09cd092bdb6
Error (code / quality):
Linting failed due to unsorted/unformatted imports and redefinitions in datadog_checks/sqlserver/xe_collection/base.py:1.
CI job: https://github.com/DataDog/integrations-core/actions/runs/30565361002/job/90948593228

Plus 41 more failing jobs not shown here.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c35b9f9 | Docs | Datadog PR Page | Give us feedback!

@azhou-datadog azhou-datadog changed the title [mysql] Prefer REFERENCES over SELECT for schema collection; make composite FK column order deterministic [mysql] Name REFERENCES in the schema collection privileges warning Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant