Skip to content

feat(telemetry): align metrics with OpenTelemetry conventions#453

Merged
yordis merged 4 commits into
masterfrom
yordis/chore-govern-metric-contract
Jul 21, 2026
Merged

feat(telemetry): align metrics with OpenTelemetry conventions#453
yordis merged 4 commits into
masterfrom
yordis/chore-govern-metric-contract

Conversation

@yordis

@yordis yordis commented Jul 20, 2026

Copy link
Copy Markdown
Member
  • Portable metric identities prevent exporter-specific interpretation and silent aggregation errors.
  • Native runtime and server instruments reduce duplicate collection paths and semantic drift.
  • Generated definitions keep metric names, units, descriptions, attributes, and instrument kinds governed by one schema.

Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com>
@cursor

cursor Bot commented Jul 20, 2026

Copy link
Copy Markdown

PR Summary

High Risk
Wide breaking rename of exported metrics and attributes affects production monitoring; large instrumentation refactor across core metrics, bootstrap, and projections with retained config keys only partially controlling what is emitted.

Overview
Breaking observability change: built-in metrics move from Prometheus-style eventstore_* names to dotted OpenTelemetry identities—mostly development trogon.eventstore.* instruments with schema-defined attributes—and a small set of official signals such as process.uptime, process.disk.io, and system.filesystem.*. Dashboards and alerts keyed on old names need to be migrated.

A local Weaver registry (otel/semconv/registry) plus split codegen (Trogon metrics/attributes vs pinned official OTel definitions) produces MetricDefinitions, TrogonAttributeNames, and OpenTelemetryMetricDefinitions. Core and projections instrumentation must reference those types; semconv/check fails if MetricsBootstrapper or projections still embed "eventstore-" literals.

Runtime wiring is updated to pass MetricDefinition through metric helpers (name, unit, description, instrument kind). Notable behavior shifts include component/projection status as one-hot UpDownCounters (1 active / 0 inactive) instead of epoch timestamps on gauges, incoming gRPC split into separate counters (failures can overlap), queue busy time as a summed counter per queue label, and duration/outcome tags using trogon.eventstore.activity.* with success/error. Custom duplicates are removed—Kestrel connection gauge, GC pause RecentMax, and many process/system trackers (CPU, memory, GC detail)—in favor of always-on System.Runtime and Microsoft.AspNetCore.Server.Kestrel meters. metricsconfig.json and MetricsConfiguration drop the removed switches; telemetry resources gain process/host/runtime attributes.

docs/diagnostics/metrics.md is rewritten as an OTel-oriented reference with configuration mapping to instrument groups.

Reviewed by Cursor Bugbot for commit 0e18038. Bugbot is set up for automated code reviews on this repo. Configure here.

@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@yordis, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 25 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 7614282a-4629-4af2-b79a-a8108561da26

📥 Commits

Reviewing files that changed from the base of the PR and between 428c983 and 0e18038.

⛔ Files ignored due to path filters (4)
  • src/TrogonEventStore.SemanticConventions/Generated/AttributeNames.g.cs is excluded by !**/generated/**
  • src/TrogonEventStore.SemanticConventions/Generated/MetricDefinitions.g.cs is excluded by !**/generated/**
  • src/TrogonEventStore.SemanticConventions/Generated/OpenTelemetryMetricDefinitions.g.cs is excluded by !**/generated/**
  • src/TrogonEventStore.SemanticConventions/Generated/TrogonAttributeNames.g.cs is excluded by !**/generated/**
📒 Files selected for processing (74)
  • .config/mise/tasks/semconv/generate
  • docs/diagnostics/metrics.md
  • otel/semconv/registry/trogon/eventstore/metrics.yaml
  • otel/semconv/templates/registry/csharp/metric-names.cs.j2
  • otel/semconv/templates/registry/csharp/open-telemetry-metric-definitions.cs.j2
  • otel/semconv/templates/registry/csharp/trogon-attribute-names.cs.j2
  • otel/semconv/templates/registry/csharp/weaver.yaml
  • src/EventStore.ClusterNode/metricsconfig.json
  • src/EventStore.Common/Configuration/MetricsConfiguration.cs
  • src/EventStore.Core.XUnit.Tests/Index/IndexStatusTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Index/IndexTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/AverageMetricTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/CacheHitsMiseesTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/CacheResourcesTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/DurationMaxTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/DurationTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/ElectionsCounterTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/IncomingGrpcCallsMetricTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/MaxTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/MetricInstrumentDefinitionTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/PersistentSubscriptionMetricsTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/ProcessMetricsTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/QueueBusyTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/QueueProcessingTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/SystemMetricsTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/TestMeterListener.cs
  • src/EventStore.Core.XUnit.Tests/OpenTelemetry/MetricNamesTests.cs
  • src/EventStore.Core.XUnit.Tests/OpenTelemetry/TelemetryMeterInstrumentationTests.cs
  • src/EventStore.Core.XUnit.Tests/OpenTelemetry/TelemetryServiceIdentityTests.cs
  • src/EventStore.Core.XUnit.Tests/Scavenge/ScavengeStatusTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Services/VNode/NodeStatusTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/TransactionLog/Checkpoint/CheckpointMetricTests.cs
  • src/EventStore.Core.XUnit.Tests/TransactionLog/Chunks/TFChunkTrackerTests.cs
  • src/EventStore.Core/ClusterVNodeStartup.cs
  • src/EventStore.Core/Diagnostics/TelemetryMeterInstrumentation.cs
  • src/EventStore.Core/Diagnostics/TelemetryServiceIdentity.cs
  • src/EventStore.Core/Index/IndexStatusTracker.cs
  • src/EventStore.Core/Metrics/ActivityStatusSubMetric.cs
  • src/EventStore.Core/Metrics/AverageMetric.cs
  • src/EventStore.Core/Metrics/CacheHitsMissesMetric.cs
  • src/EventStore.Core/Metrics/CacheResourcesMetrics.cs
  • src/EventStore.Core/Metrics/ConnectionMetric.cs
  • src/EventStore.Core/Metrics/CounterMetric.cs
  • src/EventStore.Core/Metrics/Duration.cs
  • src/EventStore.Core/Metrics/DurationMaxMetric.cs
  • src/EventStore.Core/Metrics/DurationMaxTracker.cs
  • src/EventStore.Core/Metrics/DurationMetric.cs
  • src/EventStore.Core/Metrics/GCSuspensionMetric.cs
  • src/EventStore.Core/Metrics/IncomingGrpcCallsMetric.cs
  • src/EventStore.Core/Metrics/LogicalChunkReadDistributionMetric.cs
  • src/EventStore.Core/Metrics/MaxMetric.cs
  • src/EventStore.Core/Metrics/MaxTracker.cs
  • src/EventStore.Core/Metrics/ObservableUpDownMetric.cs
  • src/EventStore.Core/Metrics/PersistentSubscriptionTracker.cs
  • src/EventStore.Core/Metrics/ProcessMetrics.cs
  • src/EventStore.Core/Metrics/QueueBusyTracker.cs
  • src/EventStore.Core/Metrics/QueueLengthTracker.cs
  • src/EventStore.Core/Metrics/QueueProcessingTracker.cs
  • src/EventStore.Core/Metrics/StatusMetric.cs
  • src/EventStore.Core/Metrics/StatusSubMetric.cs
  • src/EventStore.Core/Metrics/SummedCounterMetric.cs
  • src/EventStore.Core/Metrics/SystemMetrics.cs
  • src/EventStore.Core/MetricsBootstrapper.cs
  • src/EventStore.Core/Services/VNode/NodeStatusTracker.cs
  • src/EventStore.Core/TransactionLog/Checkpoint/CheckpointMetric.cs
  • src/EventStore.Core/TransactionLog/Chunks/TransactionFileTracker.cs
  • src/EventStore.Core/TransactionLog/Scavenging/ScavengeStatusTracker.cs
  • src/EventStore.Projections.Core.XUnit.Tests/Metrics/ProjectionMetricsTests.cs
  • src/EventStore.Projections.Core/Metrics/ProjectionTracker.cs
  • src/EventStore.Projections.Core/ProjectionsSubsystem.cs
  • src/TrogonEventStore.SemanticConventions/MetricDefinition.cs
  • src/TrogonEventStore.SemanticConventions/MetricInstrumentKind.cs
  • src/TrogonEventStore.SemanticConventions/README.md
  • src/TrogonEventStore.SemanticConventions/TrogonEventStore.SemanticConventions.csproj

Walkthrough

Adds an OpenTelemetry metric registry and generated C# MetricNames catalog, updates metric helpers and runtime wiring to consume explicit names, revises affected tests, and adds generation and hardcoded-name validation.

Changes

Metric catalog and generation

Layer / File(s) Summary
Metric registry contract
otel/semconv/registry/manifest.yaml, otel/semconv/registry/trogon/eventstore/metrics.yaml
Defines the EventStore semantic-convention registry and metric catalog across core, system, process, storage, subscription, and projection metrics.
Pinned semantic-convention generation
.config/mise/tasks/semconv/*, otel/semconv/templates/registry/csharp/*
Validates the pinned local registry and generates attribute and metric C# outputs through separate parameterized Weaver passes.
Explicit metric-name registration
src/EventStore.Core/Metrics/*.cs
Removes unit-suffix construction from metric helpers so supplied names are registered directly.
Runtime catalog adoption and validation
src/EventStore.Core/MetricsBootstrapper.cs, src/EventStore.Core/ClusterVNodeStartup.cs, src/EventStore.Projections.Core/ProjectionsSubsystem.cs, src/EventStore.Core.XUnit.Tests/**, .config/mise/tasks/semconv/check, src/TrogonEventStore.SemanticConventions/README.md
Replaces hardcoded runtime names with MetricNames constants, updates test instruments, validates catalog ordering, and rejects hardcoded built-in metric strings.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SemconvTask
  participant LocalRegistry
  participant Weaver
  participant MetricNames
  SemconvTask->>LocalRegistry: validate pinned manifest
  SemconvTask->>Weaver: generate metric conventions
  Weaver->>MetricNames: emit constants and All
  SemconvTask->>MetricNames: validate generated artifacts
Loading

Possibly related PRs

Poem

I’m a rabbit with names in a row,
From registry roots they now grow.
No hardcoded strings in the breeze,
Just catalog constants with ease.
MetricNames.All hops bright and true—
A tidy metric trail for you!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title accurately summarizes the main change: aligning telemetry metric naming with OpenTelemetry conventions.
Description check ✅ Passed The description is clearly related to the changeset and describes the telemetry naming and schema unification goals.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch yordis/chore-govern-metric-contract

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
src/EventStore.Core/ClusterVNodeStartup.cs (1)

320-339: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚖️ Poor tradeoff

Suffix-based histogram bucket selection still remains (unchanged).

This block still classifies histograms by i.Name.EndsWith("-latency-seconds")/"-seconds" string suffixes — the same "hidden suffix rule" pattern this PR's objective calls out for removal. It's unchanged here, but the migration to explicit MetricNames constants (line 308) is incomplete relative to that stated goal.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/EventStore.Core/ClusterVNodeStartup.cs` around lines 320 - 339, Replace
the suffix-based checks in the histogram bucket selection logic with explicit
comparisons against the appropriate MetricNames constants, including the
latency-seconds and seconds metrics. Preserve the existing bucket configurations
and ensure both branches continue selecting the same histogram boundaries.
otel/semconv/templates/registry/csharp/metric-names.cs.j2 (1)

10-22: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Extract the repeated name-derivation filter chain to avoid drift between const declarations and the All list.

metric.metric_name | pascal_case | regex_replace("^Eventstore", "") is duplicated verbatim at line 12 and line 20. Since line 20 must produce the exact identifier declared at line 12 to compile, any future change to this transform in only one place will break the build or silently desync the All list from the declared consts. Consider binding the result to a Jinja variable once per metric and reusing it in both loops.

Separately, please confirm pascal_case and regex_replace are filters actually provided by the pinned Weaver/MiniJinja version — Weaver's documented custom filters (prometheus_*, kebab_case_const, etc.) don't clearly list these two in what's publicly documented.

♻️ Proposed DRY refactor
 		public static class MetricNames
 		{
 {% for group in ctx %}
-{% for metric in group.metrics | sort(attribute="metric_name") %}
-		public const string {{ metric.metric_name | pascal_case | regex_replace("^Eventstore", "") }} = "{{ metric.metric_name }}";
+{% for metric in group.metrics | sort(attribute="metric_name") %}
+{% set const_name = metric.metric_name | pascal_case | regex_replace("^Eventstore", "") %}
+		public const string {{ const_name }} = "{{ metric.metric_name }}";
 {% endfor %}
 {% endfor %}

 		public static IReadOnlyList<string> All { get; } = Array.AsReadOnly(new[]
 		{
 {% for group in ctx %}
-{% for metric in group.metrics | sort(attribute="metric_name") %}
-			{{ metric.metric_name | pascal_case | regex_replace("^Eventstore", "") }},
+{% for metric in group.metrics | sort(attribute="metric_name") %}
+{% set const_name = metric.metric_name | pascal_case | regex_replace("^Eventstore", "") %}
+			{{ const_name }},
 {% endfor %}
 {% endfor %}
 		});
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@otel/semconv/templates/registry/csharp/metric-names.cs.j2` around lines 10 -
22, Centralize the metric identifier derivation used by the const declarations
and `All` list so both outputs reuse the same value and cannot drift; use a
Jinja macro or shared binding appropriate to the template scope around the
`metric.metric_name` loops. Verify that the pinned Weaver/MiniJinja version
provides `pascal_case` and `regex_replace`; if either is unavailable, replace it
with the supported equivalent while preserving the existing identifier
transformation.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/EventStore.Core.XUnit.Tests/Metrics/ProcessMetricsTests.cs`:
- Around line 67-72: Update the configuration passed to CreateDiskOpsMetric in
ProcessMetricsTests so it uses MetricsConfiguration.ProcessTracker.DiskReadOps
and DiskWrittenOps, while leaving the existing disk-bytes metric configuration
unchanged.

In `@src/EventStore.Core/MetricsBootstrapper.cs`:
- Around line 351-355: Update the expected literal for
MetricsConfiguration.ProcessTracker.GcPauseDuration in ProcessMetricsTests to
use the catalog name eventstore-gc-pause-duration-max-seconds, matching
MetricNames.GcPauseDurationMaxSeconds and production output.

---

Nitpick comments:
In `@otel/semconv/templates/registry/csharp/metric-names.cs.j2`:
- Around line 10-22: Centralize the metric identifier derivation used by the
const declarations and `All` list so both outputs reuse the same value and
cannot drift; use a Jinja macro or shared binding appropriate to the template
scope around the `metric.metric_name` loops. Verify that the pinned
Weaver/MiniJinja version provides `pascal_case` and `regex_replace`; if either
is unavailable, replace it with the supported equivalent while preserving the
existing identifier transformation.

In `@src/EventStore.Core/ClusterVNodeStartup.cs`:
- Around line 320-339: Replace the suffix-based checks in the histogram bucket
selection logic with explicit comparisons against the appropriate MetricNames
constants, including the latency-seconds and seconds metrics. Preserve the
existing bucket configurations and ensure both branches continue selecting the
same histogram boundaries.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: fbc1e4c7-bbe9-4f20-8d8f-ea83ab0a915e

📥 Commits

Reviewing files that changed from the base of the PR and between 716895e and 428c983.

⛔ Files ignored due to path filters (1)
  • src/TrogonEventStore.SemanticConventions/Generated/MetricNames.g.cs is excluded by !**/generated/**
📒 Files selected for processing (29)
  • .config/mise/tasks/semconv/check
  • .config/mise/tasks/semconv/generate
  • otel/semconv/registry/manifest.yaml
  • otel/semconv/registry/trogon/eventstore/metrics.yaml
  • otel/semconv/templates/registry/csharp/metric-names.cs.j2
  • otel/semconv/templates/registry/csharp/weaver.yaml
  • src/EventStore.Core.XUnit.Tests/Index/IndexTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/AverageMetricTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/CacheResourcesTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/DurationMaxTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/DurationTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/ElectionsCounterTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/ProcessMetricsTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/QueueBusyTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/QueueProcessingTrackerTests.cs
  • src/EventStore.Core.XUnit.Tests/Metrics/SystemMetricsTests.cs
  • src/EventStore.Core.XUnit.Tests/OpenTelemetry/MetricNamesTests.cs
  • src/EventStore.Core.XUnit.Tests/TransactionLog/Chunks/TFChunkTrackerTests.cs
  • src/EventStore.Core/ClusterVNodeStartup.cs
  • src/EventStore.Core/Metrics/AverageMetric.cs
  • src/EventStore.Core/Metrics/CacheResourcesMetrics.cs
  • src/EventStore.Core/Metrics/CounterMetric.cs
  • src/EventStore.Core/Metrics/DurationMaxMetric.cs
  • src/EventStore.Core/Metrics/DurationMetric.cs
  • src/EventStore.Core/Metrics/ProcessMetrics.cs
  • src/EventStore.Core/Metrics/SystemMetrics.cs
  • src/EventStore.Core/MetricsBootstrapper.cs
  • src/EventStore.Projections.Core/ProjectionsSubsystem.cs
  • src/TrogonEventStore.SemanticConventions/README.md

Comment thread src/EventStore.Core.XUnit.Tests/Metrics/ProcessMetricsTests.cs Outdated
Comment thread src/EventStore.Core/MetricsBootstrapper.cs Outdated
Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com>
@yordis

yordis commented Jul 20, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@yordis

yordis commented Jul 20, 2026

Copy link
Copy Markdown
Member Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com>
@yordis yordis changed the title chore(telemetry): protect built-in metric compatibility feat(telemetry): align metrics with OpenTelemetry conventions Jul 21, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e0f2dac. Configure here.

Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com>
@yordis
yordis merged commit 1f38699 into master Jul 21, 2026
36 of 38 checks passed
@yordis
yordis deleted the yordis/chore-govern-metric-contract branch July 21, 2026 02:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant