Skip to content

Isolate delta baselines for each metric reader - #6904

Merged
tim-smart merged 2 commits into
mainfrom
audit/repro-otel-metrics-multi-reader-delta
Aug 3, 2026
Merged

Isolate delta baselines for each metric reader#6904
tim-smart merged 2 commits into
mainfrom
audit/repro-otel-metrics-multi-reader-delta

Conversation

@fubhy

@fubhy fubhy commented Aug 3, 2026

Copy link
Copy Markdown
Member

Summary

When multiple registered readers collect sequentially, later readers lose measurements already consumed by earlier readers.

Important

This PR starts with focused failing reproduction tests. Add the implementation fix to this same branch; CI is expected to fail until that fix is included.

Metric readers share one delta baseline

Module: opentelemetry/OtelMetrics
Audit ID: adapters-otel-2-shared-reader-delta-baseline
Severity / confidence: high / high

What happens

When multiple registered readers collect sequentially, later readers lose measurements already consumed by earlier readers.

Why it happens

All readers share one MetricProducerImpl whose previous values and previous export time mutate after every collect.

Expected behavior

Each reader accepted by registerProducer must receive measurements for its own delta collection interval.

Relevant implementation

These links and excerpts are pinned to audit base c9b56ab507f224426ee8388dc450da447ec4715f.

View problematic code at packages/opentelemetry/src/OtelMetrics.ts:71-95
export const registerProducer = (
  self: MetricProducer,
  metricReader: LazyArg<MetricReader | Arr.NonEmptyReadonlyArray<MetricReader>>,
  options?: {
    readonly shutdownTimeout?: Duration.Input | undefined
  }
): Effect.Effect<Array<any>, never, Scope.Scope> =>
  Effect.acquireRelease(
    Effect.sync(() => {
      const reader = metricReader()
      const readers: Array<MetricReader> = Array.isArray(reader) ? reader : [reader] as any
      readers.forEach((reader) => reader.setMetricProducer(self))
      return readers
    }),
    (readers) =>
      Effect.promise(() =>
        Promise.all(
          readers.map((reader) => reader.shutdown())
        )
      ).pipe(
        Effect.ignore,
        Effect.interruptible,
        Effect.timeoutOption(options?.shutdownTimeout ?? 3000)
      )
  )

View exact lines on GitHub

View problematic code at packages/opentelemetry/src/internal/metrics.ts:46-65
  previousCounterState: Map<string, number | bigint>
  previousHistogramState: Map<string, PreviousHistogramState>
  previousFrequencyState: Map<string, Map<string, number>>
  previousSummaryState: Map<string, PreviousSummaryState>

  constructor(
    resource: Resources.Resource,
    context: Context.Context<never>,
    temporality: Metrics.TemporalityPreference = "cumulative"
  ) {
    this.resource = resource
    this.context = context
    this.temporality = temporality
    this.startTimes = new Map()
    this.startTimeNanos = currentHrTime()
    this.previousExportTimeNanos = this.startTimeNanos
    this.previousCounterState = new Map()
    this.previousHistogramState = new Map()
    this.previousFrequencyState = new Map()
    this.previousSummaryState = new Map()

View exact lines on GitHub

View problematic code at packages/opentelemetry/src/internal/metrics.ts:109-129
          if (isDelta) {
            const previousCount = this.previousCounterState.get(metricKey)
            if (previousCount !== undefined) {
              if (typeof currentCount === "bigint" && typeof previousCount === "bigint") {
                reportValue = currentCount - previousCount
                // Handle reset: if current < previous, report current value
                if (reportValue < BigInt(0)) {
                  reportValue = currentCount
                }
              } else {
                const curr = Number(currentCount)
                const prev = Number(previousCount)
                reportValue = curr - prev
                // Handle reset
                if (reportValue < 0) {
                  reportValue = curr
                }
              }
            }
            this.previousCounterState.set(metricKey, currentCount)
          }

View exact lines on GitHub

View problematic code at packages/opentelemetry/src/internal/metrics.ts:395-398
    // Update the previous export time for delta calculations
    if (isDelta) {
      this.previousExportTimeNanos = hrTimeNow
    }

View exact lines on GitHub

Reproduction

pnpm test --run packages/opentelemetry/test/OtelMetricsMultiReaderDeltaRepro.test.ts

Observed failure: Readers received [1, 0] instead of [1, 1] (1/1 failed).

Implementation handoff

The initial reproduction tests on this branch are the regression specification for the implementation fix that should follow in this PR.

  1. Start with the pinned implementation excerpts and the Why it happens analysis above.
  2. Change the implementation so it satisfies the stated Expected behavior; do not weaken or remove the reproduction assertions.
  3. Run the focused reproduction command(s) and confirm the observed failures become passing tests:
pnpm test --run packages/opentelemetry/test/OtelMetricsMultiReaderDeltaRepro.test.ts
  1. Run the affected package's existing tests, then the repository lint and type checks before requesting review.

Audit provenance

  • Audit base: c9b56ab507f224426ee8388dc450da447ec4715f
  • Reproduction base: c9b56ab507f224426ee8388dc450da447ec4715f
  • Findings: adapters-otel-2-shared-reader-delta-baseline
  • Initial patch: focused reproduction tests; implementation fix pending

Closes EFF-345

@github-project-automation github-project-automation Bot moved this to Discussion Ongoing in PR Backlog Aug 3, 2026
@changeset-bot

changeset-bot Bot commented Aug 3, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 87ea0bd

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 30 packages
Name Type
@effect/opentelemetry Patch
effect Patch
@effect/platform-browser Patch
@effect/platform-bun Patch
@effect/platform-deno Patch
@effect/platform-node-shared Patch
@effect/platform-node Patch
@effect/vitest Patch
@effect/ai-anthropic Patch
@effect/ai-openai-compat Patch
@effect/ai-openai Patch
@effect/ai-openrouter Patch
@effect/atom-react Patch
@effect/atom-solid Patch
@effect/atom-vue Patch
@effect/sql-clickhouse Patch
@effect/sql-d1 Patch
@effect/sql-libsql Patch
@effect/sql-mssql Patch
@effect/sql-mysql2 Patch
@effect/sql-pg Patch
@effect/sql-pglite Patch
@effect/sql-sqlite-bun Patch
@effect/sql-sqlite-do Patch
@effect/sql-sqlite-node Patch
@effect/sql-sqlite-react-native Patch
@effect/sql-sqlite-wasm Patch
@effect/docgen Patch
@effect/doctest Patch
@effect/openapi-generator Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@effect-slopcop effect-slopcop Bot added the 4.0 label Aug 3, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • Reproduction test for delta multi-reader baseline sharing — adds OtelMetricsMultiReaderDeltaRepro.test.ts that creates a delta-mode producer, registers two readers, increments a counter, collects from both, and asserts both report [1, 1]. The test correctly reproduces the bug: MetricProducerImpl.collect updates instance-level previousCounterState on each call, so the second reader computes its delta from already-advanced state and reports 0.

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) (Kimi K2 not used — the program covers this model; add its provider key to run your pick) | 𝕏

@fubhy fubhy changed the title Add reproduction for opentelemetry/OtelMetrics issue Isolate delta baselines for each metric reader Aug 3, 2026
@effect-slopcop effect-slopcop Bot added the bug Something isn't working label Aug 3, 2026
@fubhy fubhy added the audit Findings originating from the Effect runtime correctness audit label Aug 3, 2026
@tim-smart
tim-smart enabled auto-merge (squash) August 3, 2026 21:29

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • Added MetricProducerImpl.fork() — creates a fresh instance with independent delta-tracking state maps while sharing the same resource, context, and temporality.
  • Forked producer per reader in registerProducer — each registered MetricReader now receives its own MetricProducerImpl copy via fork(), isolating delta baselines so later readers compute deltas from their own collection interval.
  • Consolidated reproduction test into OtelMetrics.test.ts — the standalone repro file is deleted and replaced with a permanent regression test that creates a delta producer, registers two readers, increments a counter, and asserts both report [1, 1].

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) (Kimi K2 not used — the program covers this model; add its provider key to run your pick) | 𝕏

@tim-smart
tim-smart merged commit 0d83916 into main Aug 3, 2026
17 checks passed
@tim-smart
tim-smart deleted the audit/repro-otel-metrics-multi-reader-delta branch August 3, 2026 21:47
@github-project-automation github-project-automation Bot moved this from Discussion Ongoing to Done in PR Backlog Aug 3, 2026
@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Bundle Size Analysis

Generated from PR build output; treat the content below as untrusted.

File Name Current Size Previous Size Difference
basic.ts 7.06 KB 7.06 KB 0.00 KB (0.00%)
batching.ts 9.86 KB 9.86 KB 0.00 KB (0.00%)
brand.ts 6.34 KB 6.34 KB 0.00 KB (0.00%)
cache.ts 10.62 KB 10.62 KB 0.00 KB (0.00%)
config.ts 20.60 KB 20.60 KB 0.00 KB (0.00%)
differ.ts 20.20 KB 20.20 KB 0.00 KB (0.00%)
http-client.ts 21.49 KB 21.49 KB 0.00 KB (0.00%)
logger.ts 10.76 KB 10.76 KB 0.00 KB (0.00%)
metric.ts 8.99 KB 8.99 KB 0.00 KB (0.00%)
optic.ts 7.18 KB 7.18 KB 0.00 KB (0.00%)
pubsub.ts 14.90 KB 14.90 KB 0.00 KB (0.00%)
queue.ts 11.58 KB 11.58 KB 0.00 KB (0.00%)
schedule.ts 10.74 KB 10.74 KB 0.00 KB (0.00%)
schema-class.ts 19.14 KB 19.14 KB 0.00 KB (0.00%)
schema-fromJsonSchemaDocument.ts 28.96 KB 28.96 KB 0.00 KB (0.00%)
schema-representation-roundtrip.ts 25.29 KB 25.29 KB 0.00 KB (0.00%)
schema-string-transformation.ts 13.30 KB 13.30 KB 0.00 KB (0.00%)
schema-string.ts 10.94 KB 10.94 KB 0.00 KB (0.00%)
schema-template-literal.ts 15.17 KB 15.17 KB 0.00 KB (0.00%)
schema-toArbitraryLazy.ts 21.94 KB 21.94 KB 0.00 KB (0.00%)
schema-toCodeDocument.ts 24.34 KB 24.34 KB 0.00 KB (0.00%)
schema-toCodecJson.ts 19.18 KB 19.18 KB 0.00 KB (0.00%)
schema-toEquivalence.ts 19.01 KB 19.01 KB 0.00 KB (0.00%)
schema-toFormatter.ts 18.87 KB 18.87 KB 0.00 KB (0.00%)
schema-toJsonSchemaDocument.ts 22.60 KB 22.60 KB 0.00 KB (0.00%)
schema-toRepresentation.ts 19.52 KB 19.52 KB 0.00 KB (0.00%)
schema.ts 18.41 KB 18.41 KB 0.00 KB (0.00%)
stm.ts 12.54 KB 12.54 KB 0.00 KB (0.00%)
stream.ts 9.79 KB 9.79 KB 0.00 KB (0.00%)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

4.0 audit Findings originating from the Effect runtime correctness audit bug Something isn't working

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants