Skip to content

Emit a per-node clickhouse_node tag across all node-scoped ClickHouse checks - #24579

Draft
sangeetashivaji wants to merge 2 commits into
masterfrom
sangeeta.shivajirao/clickhouse-node-tag
Draft

Emit a per-node clickhouse_node tag across all node-scoped ClickHouse checks#24579
sangeetashivaji wants to merge 2 commits into
masterfrom
sangeeta.shivajirao/clickhouse-node-tag

Conversation

@sangeetashivaji

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds a uniform, additive per-node tag (clickhouse_node, value = ClickHouse hostName()) to every ClickHouse check that carries node-specific data, so metrics/samples/etc. can be sliced (and billed) per node. database_instance stays cluster-level, so existing cluster grouping/filtering is unaffected.

Changes by area:

  • utils: new node_tag() helper (reuses the existing CLUSTER_NODE_TAG = 'clickhouse_node').
  • Standard metrics (get_queries): system.replicas and system.dictionaries are now read via clusterAllReplicas and tagged per node in single-endpoint mode. system.parts is intentionally left plain — its GROUP BY is incompatible with the cluster_aware_query rewrite; per-node parts data comes from the DBM parts/merges check.
  • Query metrics (statements.py): rows are no longer merged across nodes (_merge_rows_across_nodes removed). Rows are grouped by node and emitted as one payload per node, each carrying clickhouse_node. FQT events are node-tagged too.
  • Query completions / errors / samples: payloads are split per node, each tagged clickhouse_node. ACTIVE_QUERIES_QUERY / ACTIVE_CONNECTIONS_QUERY now select hostName().
  • parts_and_merges / table_metrics (view refreshes): clickhouse_node added alongside the existing server_node: / host: tags (kept for backward compatibility).

Motivation

ClickHouse Cloud billing is moving to per-node. Today the resource primary key is database_instance, which is cluster-level, and most check data is aggregated across nodes before it leaves the Agent — so there is no reliable way to attribute metrics to a node. This makes every node-scoped check emit a consistent per-node tag.

Breaking behavior: query-metrics per-query series change from cluster-summed to per-node; cluster totals must be re-aggregated across clickhouse_node. Acceptable while the feature is in preview.

Open dependency: the per-node approach emits multiple database_monitoring_query_metrics / query_activity payloads for the same (database_instance, timestamp). This assumes the DBM intake does not dedupe on that key — needs confirmation with the dbm-metrics-processor team before merge.

Review checklist (to be filled by reviewers)

  • Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
  • Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
  • If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

… checks

ClickHouse Cloud billing is moving to per-node, but most check data is
aggregated across nodes before it leaves the Agent. This adds a uniform,
additive clickhouse_node tag (value = hostName()) to every check that carries
node-specific data, so metrics/samples can be sliced per node while
database_instance stays cluster-level.

- utils: add node_tag() helper (reuses CLUSTER_NODE_TAG)
- standard metrics: also tag system.replicas / system.dictionaries per node
- query metrics: stop merging rows across nodes; emit one payload per node
- query completions / errors / samples: split payloads per node
- parts_and_merges / view-refresh gauges: add clickhouse_node alongside existing tags

The query-metrics change is breaking: per-query series become per-node and must
be re-aggregated across clickhouse_node for cluster totals.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Major version bump
The changelog type changed or removed was used in this Pull Request, so the next release will bump major version. Please make sure this is a breaking change, or use the fixed or added type instead.

@dd-octo-sts

dd-octo-sts Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Validation Report

All 21 validations passed.

Show details
Validation Description Status
agent-reqs Verify check versions match the Agent requirements file
ci Validate CI configuration and code coverage settings
codeowners Validate every integration has a CODEOWNERS entry
config Validate default configuration files against spec.yaml
dep Verify dependency pins are consistent and Agent-compatible
http Validate integrations use the HTTP wrapper correctly
imports Validate check imports do not use deprecated modules
integration-style Validate check code style conventions
jmx-metrics Validate JMX metrics definition files and config
labeler Validate PR labeler config matches integration directories
legacy-signature Validate no integration uses the legacy Agent check signature
license-headers Validate Python files have proper license headers
licenses Validate third-party license attribution list
metadata Validate metadata.csv metric definitions
models Validate configuration data models match spec.yaml
openmetrics Validate OpenMetrics integrations disable the metric limit
package Validate Python package metadata and naming
qa-label Validate the pull request declares whether it needs QA for the next Agent release
readmes Validate README files have required sections
saved-views Validate saved view JSON file structure and fields
version Validate version consistency between package and changelog

View full run

@datadog-prod-us1-4

Copy link
Copy Markdown
Contributor

Tests  Code Coverage

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 99.53%
Overall Coverage: 94.27% (+5.96%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 702090f | Docs | Datadog PR Page | Give us feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant