fix(observability): stop exporting metrics to a collector that is not there - #138
Conversation
… there Micrometer's OTLP metrics registry is opt-out, and its URL defaults to http://localhost:4318/v1/metrics — inside a container, that container. Both services therefore exported to nowhere every minute and logged the failure, from deployment until 2026-07-29. OTLP tracing has the opposite default: no endpoint, no exporter, no noise. Metrics now match it, which is the change that stops the production symptom this increment opened on. OTLP log export is disabled too. The collector tails the json-file driver, so it would be a second copy of every line, and it fails the same silent way. Production also disables Boot's OTEL_* environment-variable mapping. The design had assumed OTEL_EXPORTER_OTLP_ENDPOINT would carry the endpoint; that mapping is a way for OrgMemory telemetry to acquire a destination nobody chose for it, and this host already runs an unrelated OpenTelemetry stack. Knowing from one file where telemetry goes is worth more than the convention. The design records the reversal rather than quietly dropping the earlier paragraph. Spans gain service.version and deployment.environment, without which a collector cannot separate builds or environments. Worker sampling goes to 1.0: it runs a low-volume batch workload, so a tenth of it leaves most indexing jobs with no trace at all. The API stays at 0.1, where request volume is the reason sampling exists. spring.ai.chat.observations.include-error-logging is declared explicitly. It was already the framework default; stating it keeps the payload posture in one readable block instead of two settings and an assumption. The API context test starts the real application and asserts the absence of the meter registry and the presence of both resource attributes. The worker has no bootable context test, so its defaults are asserted against the shipped YAML with placeholders resolved as an environment that sets nothing would resolve them. Production has not been restarted with this configuration, so "no recurring exporter warning" is still an exit criterion rather than an observation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 42 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughThe API and worker applications now explicitly configure OTLP metrics and log export, trace sampling, OpenTelemetry resource attributes, production environment-variable mapping, and Spring AI observation error logging. Tests validate runtime telemetry beans and shipped worker configuration defaults. ChangesTelemetry configuration
Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/api/src/main/resources/application.yml`:
- Around line 198-203: The OpenTelemetry deployment attribute is deprecated;
replace deployment.environment with deployment.environment.name in both
application configurations. Update the API test OrgMemoryApiContextLoadTests and
worker test TelemetryExportDefaultsTests to assert the new key, covering
apps/api/src/main/resources/application.yml lines 198-203,
apps/worker/src/main/resources/application.yml lines 140-145,
apps/api/src/test/java/com/orgmemory/api/OrgMemoryApiContextLoadTests.java lines
57-63, and
apps/worker/src/test/java/com/orgmemory/worker/observability/TelemetryExportDefaultsTests.java
lines 53-64.
In `@apps/api/src/test/java/com/orgmemory/api/OrgMemoryApiContextLoadTests.java`:
- Around line 3-14: Add the Spring Boot metrics test module dependency in
apps/api/build.gradle.kts at lines 39-43, and annotate
OrgMemoryApiContextLoadTests in
apps/api/src/test/java/com/orgmemory/api/OrgMemoryApiContextLoadTests.java at
lines 3-14 with `@AutoConfigureMetrics`; retain the existing OTLP
OtlpMeterRegistry assertion at lines 45-55 so it runs against the real
auto-configured registry.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: e1246e70-5f85-48b4-b8a8-7b8eed709db5
⛔ Files ignored due to path filters (4)
docs/increments/active/2026-07-29-observability-pipeline/design.mdis excluded by!docs/**docs/increments/active/2026-07-29-observability-pipeline/plan.mdis excluded by!docs/**docs/specs/domains/secure-graph-rag.mdis excluded by!docs/**docs/tests/domains/secure-graph-rag.mdis excluded by!docs/**
📒 Files selected for processing (7)
apps/api/build.gradle.ktsapps/api/src/main/resources/application-prod.ymlapps/api/src/main/resources/application.ymlapps/api/src/test/java/com/orgmemory/api/OrgMemoryApiContextLoadTests.javaapps/worker/src/main/resources/application-prod.ymlapps/worker/src/main/resources/application.ymlapps/worker/src/test/java/com/orgmemory/worker/observability/TelemetryExportDefaultsTests.java
📜 Review details
⏰ Context from checks skipped due to timeout. (2)
- GitHub Check: Backend · Java 25
- GitHub Check: Web · Node 24
🧰 Additional context used
📓 Path-based instructions (4)
**/*
📄 CodeRabbit inference engine (CLAUDE.md)
**/*: Always read the repository guidance and relevant sections ofARCHITECTURE.md; before changing a domain, read its specification, test-coverage document, and binding decision filenames.
Treat the repository as the engineering system of record; current repository and runtime evidence take precedence over chat or Northstar.
Readdocs/guidelines/agent-safety.mdbefore retrieval, AI, MCP, permission, upload, graph, or export work. Never commit secrets or customer data.
Files:
apps/api/src/main/resources/application-prod.ymlapps/api/build.gradle.ktsapps/api/src/main/resources/application.ymlapps/worker/src/main/resources/application-prod.ymlapps/worker/src/test/java/com/orgmemory/worker/observability/TelemetryExportDefaultsTests.javaapps/worker/src/main/resources/application.ymlapps/api/src/test/java/com/orgmemory/api/OrgMemoryApiContextLoadTests.java
**/*.{java,gradle,gradle.kts,properties,yml,yaml}
📄 CodeRabbit inference engine (CLAUDE.md)
Before using unfamiliar Spring Boot 4, Spring Modulith 2, Spring AI 2, or Gradle APIs, consult current official documentation, Context7, and the relevant project verification skill.
Files:
apps/api/src/main/resources/application-prod.ymlapps/api/build.gradle.ktsapps/api/src/main/resources/application.ymlapps/worker/src/main/resources/application-prod.ymlapps/worker/src/test/java/com/orgmemory/worker/observability/TelemetryExportDefaultsTests.javaapps/worker/src/main/resources/application.ymlapps/api/src/test/java/com/orgmemory/api/OrgMemoryApiContextLoadTests.java
**/*.{java,gradle,gradle.kts}
📄 CodeRabbit inference engine (CLAUDE.md)
Use the testing harness; a terminating clean test is the JVM context gate, and
bootRunis not verification. IDE inspection applies only to edited backend Java.
Files:
apps/api/build.gradle.ktsapps/worker/src/test/java/com/orgmemory/worker/observability/TelemetryExportDefaultsTests.javaapps/api/src/test/java/com/orgmemory/api/OrgMemoryApiContextLoadTests.java
**/*.{java,sql}
📄 CodeRabbit inference engine (CLAUDE.md)
Keep
ddl-auto=validateand pair every persisted-model change with a Flyway migration.
Files:
apps/worker/src/test/java/com/orgmemory/worker/observability/TelemetryExportDefaultsTests.javaapps/api/src/test/java/com/orgmemory/api/OrgMemoryApiContextLoadTests.java
🔇 Additional comments (5)
apps/api/src/main/resources/application.yml (1)
14-16: LGTM!apps/worker/src/main/resources/application.yml (1)
25-27: LGTM!apps/api/src/main/resources/application-prod.yml (1)
71-76: 🩺 Stability & AvailabilityVerify explicit production OTLP destinations after disabling OTEL environment mapping. Spring Boot will no longer consume
OTEL_*exporter variables, so confirm both production profiles define equivalentmanagement...endpoints and exporter settings. (docs.spring.io)
apps/api/src/main/resources/application-prod.yml#L71-L76: verify the API collector configuration.apps/worker/src/main/resources/application-prod.yml#L38-L43: verify the worker collector configuration.Source: MCP tools
apps/api/src/test/java/com/orgmemory/api/OrgMemoryApiContextLoadTests.java (1)
37-43: LGTM!apps/worker/src/test/java/com/orgmemory/worker/observability/TelemetryExportDefaultsTests.java (1)
1-51: LGTM!Also applies to: 67-87
| import static org.junit.jupiter.api.Assertions.assertNotNull; | ||
| import static org.junit.jupiter.api.Assertions.assertTrue; | ||
| import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get; | ||
| import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status; | ||
|
|
||
| import io.micrometer.registry.otlp.OtlpMeterRegistry; | ||
| import io.opentelemetry.api.common.AttributeKey; | ||
| import io.opentelemetry.api.common.Attributes; | ||
| import io.opentelemetry.sdk.resources.Resource; | ||
| import org.junit.jupiter.api.Test; | ||
| import org.springframework.beans.factory.annotation.Autowired; | ||
| import org.springframework.context.ApplicationContext; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate guidance files =="
git ls-files | rg '(^|/)ARCHITECTURE\.md$|docs/guidelines/agent-safety\.md$|binding|specification|test-coverage' || true
echo "== target files =="
fd -a 'OrgMemoryApiContextLoadTests.java|build.gradle.kts' . | sed 's#^\./##'
echo "== target source =="
cat -n apps/api/src/test/java/com/orgmemory/api/OrgMemoryApiContextLoadTests.java | sed -n '1,120p'
echo "== target build excerpt =="
cat -n apps/api/build.gradle.kts | sed -n '1,120p'
echo "== metrics-related content in test and build =="
rg -n 'Metrics|MeterRegistry|OtlpMeterRegistry|AutoConfigureMetrics|micrometer|opentelemetry|otel|export' apps/api/src/test apps/api/build.gradle.kts README.md ARCHITECTURE.md || trueRepository: kl3inIT/OrgMemory
Length of output: 10101
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== root build files excerpt =="
cat -n build.gradle.kts | sed -n '1,240p'
echo "== api gradle plugin files =="
for f in $(git ls-files '**/orgmemory*-gradle-plugin*' '**/*spring-boot*convention*' '**/*telemetry*' '**/*metrics*' 2>/dev/null || true); do
echo "--- $f"
wc -l "$f"
sed -n '1,200p' "$f" | rg -n 'spring-boot|Boot|micrometer|otlp|OpenTelemetry|opentelemetry|AutoConfigureMetrics|spring-boot-micrometer' || true
done
echo "== search for metrics test usage in repository =="
rg -n 'AutoConfigureMetrics|spring-boot-micrometer-metrics-test|micrometer-metrics-test|noMetricsExporter|OtlpMeterRegistry|OTLP|opentelemetry.exporter' . --glob '!build/**' --glob '!build-logic/build' || trueRepository: kl3inIT/OrgMemory
Length of output: 8734
🌐 Web query:
Spring Boot @SpringBootTestnon in-memory registry suppressed@AutoConfigureMetrics @EnableAutoConfiguration documentation
💡 Result:
When using @SpringBootTest in Spring Boot, the framework intentionally defaults to only configuring an in-memory MeterRegistry [1][2]. Other non-in-memory meter registries (such as Prometheus, Datadog, etc.) are not auto-configured, even if their dependencies are present on the classpath [1][2]. If you need to prevent auto-configuration or control metrics behavior during tests, you can use the following approaches: 1. Use standard exclusion mechanisms: To suppress specific auto-configuration classes, you can use the exclude attribute on @SpringBootApplication or @EnableAutoConfiguration, or define the spring.autoconfigure.exclude property [3][4]. Note that @SpringBootTest does not provide a direct exclude attribute, so for test-specific exclusions, you should use properties in the @SpringBootTest annotation [3]: @SpringBootTest(properties = "spring.autoconfigure.exclude=org.springframework.boot.actuate.autoconfigure.metrics.export.prometheus.PrometheusMetricsExportAutoConfiguration") 2. Disable specific metric exports via properties: You can disable specific exporters by setting the appropriate management.metrics.export.enabled properties to false [5][6][7]: management.defaults.metrics.export.enabled=false 3. Interaction with @AutoConfigureMetrics: The @AutoConfigureMetrics annotation is intended to enable metrics auto-configuration in sliced tests (or to force them in integration tests) [1][8]. In sliced tests, it configures an in-memory MeterRegistry and does not support exporting to external backends [1][2]. If you are explicitly using @AutoConfigureMetrics in a test and want to suppress the registry behavior, you should remove the annotation or override the registry beans in your test configuration [1][9]. 4. Global Registry: Spring Boot automatically prevents the use of the Micrometer global registry in tests by default to avoid pinning cached test contexts [10]. You can explicitly control this with the management.metrics.use-global-registry property [6][7]. In summary, you do not typically need to "suppress" non-in-memory registries in @SpringBootTest because they are already excluded by default [1][2]. If they are appearing, it is likely due to explicit configuration or manual @Import imports. If you need to stop even the in-memory registry, use the properties mentioned in point 2 to disable metrics export entirely [5][7].
Citations:
- 1: https://docs.spring.io/spring-boot/reference/testing/spring-boot-applications.html
- 2: https://docs.spring.io/spring-boot/4.1/reference/testing/spring-boot-applications.html
- 3: Support excludes on @SpringBootTest spring-projects/spring-boot#15068
- 4: https://docs.spring.io/spring-boot/reference/using/auto-configuration.html
- 5: Add support for easily disabling metrics export spring-projects/spring-boot#21658
- 6: https://docs.spring.io/spring-boot/reference/actuator/metrics.html
- 7: https://docs.spring.io/spring-boot/3.5/reference/actuator/metrics.html
- 8: https://docs.spring.io/spring-boot/api/java/org/springframework/boot/micrometer/metrics/test/autoconfigure/AutoConfigureMetrics.html
- 9: https://docs.enterprise.spring.io/spring-boot/reference/testing/spring-boot-applications.html
- 10: spring-projects/spring-boot@c61ff15
Use the Spring Boot metrics test harness for this OTLP-registy assertion. @SpringBootTest only auto-configures an in-memory MeterRegistry by default, so this assertion cannot exercise the real OtlpMeterRegistry auto-configuration. Add @AutoConfigureMetrics and the Spring Boot metrics test module, then keep the assertion.
📍 Affects 2 files
apps/api/src/test/java/com/orgmemory/api/OrgMemoryApiContextLoadTests.java#L3-L14(this comment)apps/api/src/test/java/com/orgmemory/api/OrgMemoryApiContextLoadTests.java#L45-L55apps/api/build.gradle.kts#L39-L43
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/api/src/test/java/com/orgmemory/api/OrgMemoryApiContextLoadTests.java`
around lines 3 - 14, Add the Spring Boot metrics test module dependency in
apps/api/build.gradle.kts at lines 39-43, and annotate
OrgMemoryApiContextLoadTests in
apps/api/src/test/java/com/orgmemory/api/OrgMemoryApiContextLoadTests.java at
lines 3-14 with `@AutoConfigureMetrics`; retain the existing OTLP
OtlpMeterRegistry assertion at lines 45-55 so it runs against the real
auto-configured registry.
Sources: Coding guidelines, MCP tools
`deployment.environment` is deprecated in the semantic conventions; `io.opentelemetry.semconv.DeploymentAttributes` 1.41.1, which is what resolves here, exposes only `DEPLOYMENT_ENVIRONMENT_NAME`. A collector keying dashboards on the current name would have seen nothing. Also records in the context test why it needs no `@AutoConfigureMetrics`, since that is a reasonable thing to ask of an assertion about an exporter bean. The customizer that disables metrics export under test ships in spring-boot-micrometer-metrics-test, which is not on this module's test classpath, so the real auto-configuration runs. Confirmed by flipping management.otlp.metrics.export.enabled to true, which fails the assertion — without that check the test could have been passing vacuously. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Both findings looked at directly rather than taken on trust.
Rather than argue it, I checked: flipping So the assertion is live, not vacuous. Adding |
1 participant
Phase 1 of the observability pipeline increment. This is the change that stops the symptom the increment opened on.
The bug
Micrometer's OTLP metrics registry is opt-out, and its URL defaults to
http://localhost:4318/v1/metrics— inside a container, that container. Both services exported to nowhere every minute and logged the failure, from deployment until 2026-07-29:OTLP tracing has the opposite default — no endpoint, no exporter, no noise. Metrics now match it.
Property names were taken from each module's
spring-configuration-metadata.jsonrather than from memory, since Boot 4.1 moved several of them.Also in this change
OTLP log export off. The collector tails the
json-filedriver, so it would be a second copy of every line — and it fails the same silent way.map-environment-variables: falsein production. This reverses a paragraph in the increment's own design, which assumedOTEL_EXPORTER_OTLP_ENDPOINTwould carry the endpoint. That mapping is a way for OrgMemory telemetry to acquire a destination nobody chose for it, and this host already runs an unrelated OpenTelemetry stack. The design records the reversal rather than quietly dropping the earlier text.Resource attributes.
service.versionanddeployment.environment; without them a collector cannot separate builds or environments.Worker sampling to 1.0. A low-volume batch workload sampled at a tenth leaves most indexing jobs with no trace at all. The API stays at 0.1, where request volume is the reason sampling exists.
include-error-logging: falsedeclared explicitly. Already the framework default; stating it keeps the payload posture in one readable block.Proof
The API context test starts the real application and asserts no
OtlpMeterRegistrybean exists and that both resource attributes reach the OpenTelemetryResource— which also proves the dotted map keys bind. The worker has no bootable context test, so its defaults are asserted against the shipped YAML with placeholders resolved as an environment that sets nothing would resolve them.What is not proven
Production has not been restarted with this configuration. "No recurring exporter warning" remains an exit criterion, not an observation.
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Bug Fixes