Skip to content

Fix mysql.index.reads/updates/deletes submitted as gauge instead of monotonic_count#24125

Merged
eric-weaver merged 5 commits into
masterfrom
pierreln-dd/SDBM-2685-fix-index-metrics-monotonic-count
Jun 22, 2026
Merged

Fix mysql.index.reads/updates/deletes submitted as gauge instead of monotonic_count#24125
eric-weaver merged 5 commits into
masterfrom
pierreln-dd/SDBM-2685-fix-index-metrics-monotonic-count

Conversation

@pierreln-dd

Copy link
Copy Markdown
Contributor

What does this PR do?

QUERY_INDEX_USAGE in index_metrics.py collects count_read, count_update, and count_delete from performance_schema.table_io_waits_summary_by_index_usage. These are cumulative counters, but they were submitted with 'type': 'gauge', causing mysql.index.reads, mysql.index.updates, and mysql.index.deletes to report raw monotonically increasing totals rather than per-interval counts.

This PR changes the metric type to monotonic_count, consistent with how other cumulative performance_schema counters are handled in this check (e.g. mysql.innodb.deadlocks, mysql.performance.wait_event.count).

Motivation

Customer escalation (SDBM-2685): after upgrading from MariaDB 10.6 to 11.4, mysql.index.reads/updates/deletes appeared as continuously increasing values in dashboards. Root cause confirmed in code and reproduced in staging (datad0g.com): all MySQL/MariaDB/Aurora hosts with performance_schema enabled show the same monotonically increasing pattern.

The bug affects all agents ≥ 7.63.0 (when index_metrics shipped in integration 14.5.0). It is not MariaDB-specific — confirmed present on Aurora MySQL in staging as well.

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
  • If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

…gauge

count_read/count_update/count_delete in performance_schema.table_io_waits_summary_by_index_usage
are cumulative counters. Submitting them as gauge caused mysql.index.reads/updates/deletes to
show as monotonically increasing raw totals rather than delta counts per collection interval.
@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Jun 22, 2026

Copy link
Copy Markdown

Tests  Code Coverage

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 90.30% (+2.40%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c573510 | Docs | Datadog PR Page | Give us feedback!

pierreln-dd and others added 2 commits June 22, 2026 16:09
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…metrics

monotonic_count submissions map to count in the Datadog backend.
assert_metrics_using_metadata with check_submission_type=True applies
METRIC_TYPE_SUBMISSION_TO_BACKEND_MAP before comparing, so metadata.csv
must use count not monotonic_count.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@pierreln-dd pierreln-dd added the qa/skip-qa Automatically skip this PR for the next QA label Jun 22, 2026
…revents emission

monotonic_count metrics with collection_interval=300 are never emitted in E2E:
check_rate=True runs the check twice but the second run skips the query
(300s not elapsed), so no delta is flushed. Use at_least=0 in E2E mode.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@dd-octo-sts

dd-octo-sts Bot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Validation Report

All 21 validations passed.

Show details
Validation Description Status
agent-reqs Verify check versions match the Agent requirements file
ci Validate CI configuration and code coverage settings
codeowners Validate every integration has a CODEOWNERS entry
config Validate default configuration files against spec.yaml
dep Verify dependency pins are consistent and Agent-compatible
http Validate integrations use the HTTP wrapper correctly
imports Validate check imports do not use deprecated modules
integration-style Validate check code style conventions
jmx-metrics Validate JMX metrics definition files and config
labeler Validate PR labeler config matches integration directories
legacy-signature Validate no integration uses the legacy Agent check signature
license-headers Validate Python files have proper license headers
licenses Validate third-party license attribution list
metadata Validate metadata.csv metric definitions
models Validate configuration data models match spec.yaml
openmetrics Validate OpenMetrics integrations disable the metric limit
package Validate Python package metadata and naming
qa-label Validate the pull request declares whether it needs QA for the next Agent release
readmes Validate README files have required sections
saved-views Validate saved view JSON file structure and fields
version Validate version consistency between package and changelog

View full run

@eric-weaver eric-weaver enabled auto-merge June 22, 2026 16:54
@eric-weaver eric-weaver added this pull request to the merge queue Jun 22, 2026
Merged via the queue into master with commit dcee845 Jun 22, 2026
88 of 92 checks passed
@eric-weaver eric-weaver deleted the pierreln-dd/SDBM-2685-fix-index-metrics-monotonic-count branch June 22, 2026 16:55
@dd-octo-sts dd-octo-sts Bot added this to the 7.82.0 milestone Jun 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants