fix(observability): close the payload bypasses the telemetry boundary did not cover - #137
Merged
Merged
Conversation
Production has logged an OTLP metrics export failure every minute since 2026-07-25. The cause is a transitive default: the OpenTelemetry starter resolves micrometer-registry-otlp, whose export is opt-out and defaults to a localhost URL nothing listens on. OTLP tracing behaves oppositely and is not involved. Scoping that work surfaced a larger finding. A proposal to replace the flat payload-free rule with configurable tiers went through an independent architecture challenge, which rejected it and produced two verified bypasses of the boundary the runbook claims: Spring AI provider libraries concatenate the prompt into WARN logs that log-prompt=false does not reach, and the Micrometer OTel bridge records exception events and messages on error spans. Both are live paths carrying ACL-scoped evidence. The policy stays unchanged; the repository claims more than it enforces. Closing those bypasses is phase 0, ahead of the pipeline work. The challenge brief and verdict are kept with the increment because their working location is not versioned. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`spring.ai.chat.observations.log-prompt: false` governs Spring AI's own observation handlers. It does not reach the client libraries, which build their own WARN messages by concatenating the prompt or the response content: org/springframework/ai/openai/OpenAiChatModel.java:222 org/springframework/ai/anthropic/AnthropicChatModel.java:554 org/springframework/ai/anthropic/AnthropicChatModel.java:1219 org/springframework/ai/anthropic/AnthropicChatModel.java:1018 OrgMemory prompts carry the query, the grounded context and chunk content, so those lines put customer text into a log the deployment retains on the host. Production runs at root INFO, so nothing was suppressing them. Every one of those call sites is guarded by isWarnEnabled(), so pinning the two packages above WARN stops the message from being built rather than only from being printed. The pin lives in the base configuration, not the production profile, because a development database holds real uploaded documents too, and it is deliberately not environment-overridable: the payload-free boundary is not a per-deployment setting. The tests read the shipped YAML rather than a copy of it, and fail if a profile lowers either package back to WARN. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Micrometer's OpenTelemetry bridge ends a failed span by calling recordException(throwable) and setStatus(ERROR, throwable.getMessage()), so the message and the stack trace travel with the span. An OrgMemory exception can be raised while holding a query, an evidence chunk or a provider response body, and nothing upstream can prove otherwise. The production hardening runbook already promises that error spans carry no exception event or stack trace; until now nothing enforced it, and the release gate could not have passed. ExceptionSanitizingSpanExporter wraps every exporter and drops all event attributes except exception.type, which is a class name fixed by the source, and clears the status description. Attribute counts keep their original values so a stripped attribute still reports as dropped rather than as never recorded. Micrometer's SpanFilter is the natural hook for the events but cannot reach the status description, which DelegatingSpanData does not expose as mutable. Wrapping the exporter covers both and runs after every filter, making it the last gate before egress. Spring Boot contributes its own SpanExporters only when one is missing, so declaring the collection ahead of it replaces every exporter with a wrapped copy; the ordering itself is asserted, because a wrong beforeName would silently hand the unwrapped collection back. There is no toggle. A payload guard a deployment can switch off is not a guard. Span attributes are deliberately left alone: their allowlist is a wider question than exception handling and belongs with the whole-export test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every producer wraps emit() in a catch that discards the failure, which is the right call — an observability backend must never decide whether a retrieval or an indexing job succeeds. The cost was that a sink broken since startup looked exactly like a sink with nothing to report, across five call sites in GraphRagKnowledgeRetrievalService and one in GraphIndexingProcessor. FailureTolerantGraphRagEventSink keeps the absorption and adds the signal that was missing: a running count, the type of the most recent failure, and one log line each time the failure changes kind, so a permanently broken sink cannot flood the log at event rate while one that starts failing differently still says so. Only class names are recorded. A telemetry backend's exception message can quote the request it failed to send, so the message and the stack trace stay out on the same grounds as any other payload. Logging goes through System.Logger, so components/graph-rag-core keeps its property of having no runtime dependencies; Boot's JUL bridge routes it into the application log. The wrapper is applied where the composite is built, so production always gets it. The existing call-site catches stay as defence in depth for anyone constructing the service with a raw sink. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The semantic port came from LightRAG, so its tracing was read from the pinned v1.5.4 checkout rather than from recollection. An earlier note in this increment said it has no observability; that was wrong. It has exactly one integration: lightrag/llm/openai.py swaps openai.AsyncOpenAI for langfuse.openai.AsyncOpenAI when both Langfuse keys are set, which captures requests and responses verbatim. No other provider binding is traced, and there is no masking, no OpenTelemetry, no metrics and no spans over its own stages. The upstream therefore does what this increment's decision rejects, through a vendor drop-in rather than a designed boundary — which supports the finding that "no comparable system chose never" reflects an absence of deliberation. The comparison also surfaced a gap on this side. Stage declares fourteen values and production emits ten: PARSE, CHUNK, GLEAN and GENERATE have no producer outside tests, so a stage-grouped dashboard would carry four empty series and no parsing, chunking, gleaning or generation latency. Deletion and rebuild are absent from the enum altogether, though LightRAG tracks that path and the hardening runbook requires a drill for it. Both are now phase 2 work. Phase 0 is recorded as code-complete apart from the production log search, which needs the owner's go-ahead because it may surface customer content. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The spec's source paths cover components/graph-rag-core, integrations/graph-rag-*, the worker graph package and core/knowledge, all of which moved in this change, so reconciliation belongs here rather than at the end of the increment. Records the span sanitizer, the failure-tolerant sink wrapper and the provider logger pin as current behaviour, and states plainly what they do not cover: span attributes are unfiltered, and no test asserts the whole export against the allowlist, so "payload-free" is enforced at named points rather than proven end to end. Also records that Stage declares fourteen values while production emits ten, and that deletion and rebuild have no stage at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The provider logging fix is preventive, not remedial. The project owner confirmed on 2026-07-29 that the ZM deployment is a proof of concept with no real users and no customer data, so the retained logs hold no exposure to scope and the planned production log search has nothing to find. Recorded as an attributable statement about today's deployment rather than about the design, because the same code against a deployment holding real evidence would have been writing customer text to the host since it shipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Warning Review limit reached
Next review available in: 18 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (3)
📒 Files selected for processing (11)
📝 WalkthroughWalkthroughAdds Spring AI provider logging boundaries, failure-tolerant GraphRAG event sinks, and OpenTelemetry span export sanitization. Configurations, auto-configuration wiring, and tests validate each observability behavior. ChangesProvider logging boundaries
Failure-tolerant GraphRAG sinks
Span export sanitization
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant GraphRagEventSink
participant FailureTolerantGraphRagEventSink
participant DelegateSink
GraphRagEventSink->>FailureTolerantGraphRagEventSink: failureTolerant(sink)
FailureTolerantGraphRagEventSink->>DelegateSink: emit(event)
DelegateSink-->>FailureTolerantGraphRagEventSink: success or RuntimeException
FailureTolerantGraphRagEventSink-->>GraphRagEventSink: complete or record swallowed failure
sequenceDiagram
participant SdkTracerProvider
participant ExceptionSanitizingSpanExporter
participant SpanExporter
SdkTracerProvider->>ExceptionSanitizingSpanExporter: export(spanData)
ExceptionSanitizingSpanExporter->>ExceptionSanitizingSpanExporter: sanitize events and status
ExceptionSanitizingSpanExporter->>SpanExporter: export(sanitizedSpanData)
SpanExporter-->>SdkTracerProvider: CompletableResultCode
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
⚔️ Resolve merge conflicts 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…load-boundary # Conflicts: # docs/roadmap.md
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/api/src/main/resources/application.yml`:
- Around line 183-193: Enforce the provider logger boundary at runtime by
validating the resolved levels for org.springframework.ai.openai and
org.springframework.ai.anthropic, rejecting startup when either effective level
is below ERROR despite configuration overrides. Apply the corresponding
configuration/validation in apps/api/src/main/resources/application.yml (lines
183-193) and apps/worker/src/main/resources/application.yml (lines 118-128).
Extend ProviderLoggingBoundaryTests in
apps/api/src/test/java/com/orgmemory/api/observability/ProviderLoggingBoundaryTests.java
(lines 39-53) and
apps/worker/src/test/java/com/orgmemory/worker/observability/ProviderLoggingBoundaryTests.java
(lines 39-53) to set higher-precedence WARN overrides and verify they are
rejected.
In
`@apps/api/src/test/java/com/orgmemory/api/observability/ProviderLoggingBoundaryTests.java`:
- Around line 63-75: Update ProviderLoggingBoundaryTests in both API and worker
copies: recognize profile files ending in .yml or .yaml, and match provider
logger entries explicitly so child loggers such as
org.springframework.ai.openai.api.requests are validated even when a parent
logger has a stricter level. Apply the corresponding changes at the listed
profile-discovery and logger-validation sites in both files.
In
`@components/graph-rag-core/src/main/java/com/orgmemory/graphrag/observability/FailureTolerantGraphRagEventSink.java`:
- Around line 61-70: Update FailureTolerantGraphRagEventSink’s failure-reporting
state so logging is rate-limited per failure type rather than only when it
differs from lastFailureType(). Retain lastFailureType() as the latest observed
type, use a bounded policy to prevent unbounded state, and add coverage for an
A→B→A sequence ensuring recurring types are suppressed within the limit.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 389d1bfe-8859-4d34-8e35-e52825f1193e
⛔ Files ignored due to path filters (7)
docs/increments/active/2026-07-29-observability-pipeline/challenge-brief.mdis excluded by!docs/**docs/increments/active/2026-07-29-observability-pipeline/challenge-verdict.mdis excluded by!docs/**docs/increments/active/2026-07-29-observability-pipeline/design.mdis excluded by!docs/**docs/increments/active/2026-07-29-observability-pipeline/plan.mdis excluded by!docs/**docs/roadmap.mdis excluded by!docs/**docs/specs/domains/secure-graph-rag.mdis excluded by!docs/**docs/tests/domains/secure-graph-rag.mdis excluded by!docs/**
📒 Files selected for processing (15)
apps/api/src/main/resources/application.ymlapps/api/src/test/java/com/orgmemory/api/observability/ProviderLoggingBoundaryTests.javaapps/worker/src/main/java/com/orgmemory/worker/graph/GraphIndexingProcessor.javaapps/worker/src/main/resources/application.ymlapps/worker/src/test/java/com/orgmemory/worker/observability/ProviderLoggingBoundaryTests.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/observability/FailureTolerantGraphRagEventSink.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/observability/GraphRagEventSink.javacomponents/graph-rag-core/src/test/java/com/orgmemory/graphrag/observability/FailureTolerantGraphRagEventSinkTests.javacore/src/main/java/com/orgmemory/core/knowledge/GraphRagKnowledgeRetrievalConfiguration.javaintegrations/graph-rag-observability/build.gradle.ktsintegrations/graph-rag-observability/src/main/java/com/orgmemory/integrations/graphrag/observability/ExceptionSanitizingSpanExporter.javaintegrations/graph-rag-observability/src/main/java/com/orgmemory/integrations/graphrag/observability/SpanExportSanitizationAutoConfiguration.javaintegrations/graph-rag-observability/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.importsintegrations/graph-rag-observability/src/test/java/com/orgmemory/integrations/graphrag/observability/ExceptionSanitizingSpanExporterTests.javaintegrations/graph-rag-observability/src/test/java/com/orgmemory/integrations/graphrag/observability/SpanExportSanitizationAutoConfigurationTests.java
📜 Review details
🧰 Additional context used
📓 Path-based instructions (5)
**/*
📄 CodeRabbit inference engine (CLAUDE.md)
**/*: Always read the repository guidance and relevant sections ofARCHITECTURE.md; before changing a domain, read its specification, test-coverage document, and binding decision filenames.
Treat the repository as the engineering system of record; current repository and runtime evidence take precedence over chat or Northstar.
Readdocs/guidelines/agent-safety.mdbefore retrieval, AI, MCP, permission, upload, graph, or export work. Never commit secrets or customer data.
Files:
components/graph-rag-core/src/main/java/com/orgmemory/graphrag/observability/FailureTolerantGraphRagEventSink.javaintegrations/graph-rag-observability/src/main/java/com/orgmemory/integrations/graphrag/observability/SpanExportSanitizationAutoConfiguration.javaintegrations/graph-rag-observability/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.importsapps/api/src/main/resources/application.ymlcore/src/main/java/com/orgmemory/core/knowledge/GraphRagKnowledgeRetrievalConfiguration.javaintegrations/graph-rag-observability/build.gradle.ktsintegrations/graph-rag-observability/src/test/java/com/orgmemory/integrations/graphrag/observability/SpanExportSanitizationAutoConfigurationTests.javacomponents/graph-rag-core/src/test/java/com/orgmemory/graphrag/observability/FailureTolerantGraphRagEventSinkTests.javaapps/api/src/test/java/com/orgmemory/api/observability/ProviderLoggingBoundaryTests.javaapps/worker/src/main/java/com/orgmemory/worker/graph/GraphIndexingProcessor.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/observability/GraphRagEventSink.javaapps/worker/src/test/java/com/orgmemory/worker/observability/ProviderLoggingBoundaryTests.javaapps/worker/src/main/resources/application.ymlintegrations/graph-rag-observability/src/test/java/com/orgmemory/integrations/graphrag/observability/ExceptionSanitizingSpanExporterTests.javaintegrations/graph-rag-observability/src/main/java/com/orgmemory/integrations/graphrag/observability/ExceptionSanitizingSpanExporter.java
**/*.{java,gradle,gradle.kts,properties,yml,yaml}
📄 CodeRabbit inference engine (CLAUDE.md)
Before using unfamiliar Spring Boot 4, Spring Modulith 2, Spring AI 2, or Gradle APIs, consult current official documentation, Context7, and the relevant project verification skill.
Files:
components/graph-rag-core/src/main/java/com/orgmemory/graphrag/observability/FailureTolerantGraphRagEventSink.javaintegrations/graph-rag-observability/src/main/java/com/orgmemory/integrations/graphrag/observability/SpanExportSanitizationAutoConfiguration.javaapps/api/src/main/resources/application.ymlcore/src/main/java/com/orgmemory/core/knowledge/GraphRagKnowledgeRetrievalConfiguration.javaintegrations/graph-rag-observability/build.gradle.ktsintegrations/graph-rag-observability/src/test/java/com/orgmemory/integrations/graphrag/observability/SpanExportSanitizationAutoConfigurationTests.javacomponents/graph-rag-core/src/test/java/com/orgmemory/graphrag/observability/FailureTolerantGraphRagEventSinkTests.javaapps/api/src/test/java/com/orgmemory/api/observability/ProviderLoggingBoundaryTests.javaapps/worker/src/main/java/com/orgmemory/worker/graph/GraphIndexingProcessor.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/observability/GraphRagEventSink.javaapps/worker/src/test/java/com/orgmemory/worker/observability/ProviderLoggingBoundaryTests.javaapps/worker/src/main/resources/application.ymlintegrations/graph-rag-observability/src/test/java/com/orgmemory/integrations/graphrag/observability/ExceptionSanitizingSpanExporterTests.javaintegrations/graph-rag-observability/src/main/java/com/orgmemory/integrations/graphrag/observability/ExceptionSanitizingSpanExporter.java
**/*.{java,sql}
📄 CodeRabbit inference engine (CLAUDE.md)
Keep
ddl-auto=validateand pair every persisted-model change with a Flyway migration.
Files:
components/graph-rag-core/src/main/java/com/orgmemory/graphrag/observability/FailureTolerantGraphRagEventSink.javaintegrations/graph-rag-observability/src/main/java/com/orgmemory/integrations/graphrag/observability/SpanExportSanitizationAutoConfiguration.javacore/src/main/java/com/orgmemory/core/knowledge/GraphRagKnowledgeRetrievalConfiguration.javaintegrations/graph-rag-observability/src/test/java/com/orgmemory/integrations/graphrag/observability/SpanExportSanitizationAutoConfigurationTests.javacomponents/graph-rag-core/src/test/java/com/orgmemory/graphrag/observability/FailureTolerantGraphRagEventSinkTests.javaapps/api/src/test/java/com/orgmemory/api/observability/ProviderLoggingBoundaryTests.javaapps/worker/src/main/java/com/orgmemory/worker/graph/GraphIndexingProcessor.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/observability/GraphRagEventSink.javaapps/worker/src/test/java/com/orgmemory/worker/observability/ProviderLoggingBoundaryTests.javaintegrations/graph-rag-observability/src/test/java/com/orgmemory/integrations/graphrag/observability/ExceptionSanitizingSpanExporterTests.javaintegrations/graph-rag-observability/src/main/java/com/orgmemory/integrations/graphrag/observability/ExceptionSanitizingSpanExporter.java
**/*.{java,gradle,gradle.kts}
📄 CodeRabbit inference engine (CLAUDE.md)
Use the testing harness; a terminating clean test is the JVM context gate, and
bootRunis not verification. IDE inspection applies only to edited backend Java.
Files:
components/graph-rag-core/src/main/java/com/orgmemory/graphrag/observability/FailureTolerantGraphRagEventSink.javaintegrations/graph-rag-observability/src/main/java/com/orgmemory/integrations/graphrag/observability/SpanExportSanitizationAutoConfiguration.javacore/src/main/java/com/orgmemory/core/knowledge/GraphRagKnowledgeRetrievalConfiguration.javaintegrations/graph-rag-observability/build.gradle.ktsintegrations/graph-rag-observability/src/test/java/com/orgmemory/integrations/graphrag/observability/SpanExportSanitizationAutoConfigurationTests.javacomponents/graph-rag-core/src/test/java/com/orgmemory/graphrag/observability/FailureTolerantGraphRagEventSinkTests.javaapps/api/src/test/java/com/orgmemory/api/observability/ProviderLoggingBoundaryTests.javaapps/worker/src/main/java/com/orgmemory/worker/graph/GraphIndexingProcessor.javacomponents/graph-rag-core/src/main/java/com/orgmemory/graphrag/observability/GraphRagEventSink.javaapps/worker/src/test/java/com/orgmemory/worker/observability/ProviderLoggingBoundaryTests.javaintegrations/graph-rag-observability/src/test/java/com/orgmemory/integrations/graphrag/observability/ExceptionSanitizingSpanExporterTests.javaintegrations/graph-rag-observability/src/main/java/com/orgmemory/integrations/graphrag/observability/ExceptionSanitizingSpanExporter.java
core/src/main/java/com/orgmemory/core/{authorization,knowledge,permission}/**/*.java
⚙️ CodeRabbit configuration file
core/src/main/java/com/orgmemory/core/{authorization,knowledge,permission}/**/*.java: Treat PostgreSQL ACL evidence as canonical and OpenFGA as the relationship
authorization decision point. Authorization must fail closed. Filtering
must happen before ranking, LIMIT, graph traversal, answer generation,
export, and citation rendering. Flag metadata or timing leak paths.
Files:
core/src/main/java/com/orgmemory/core/knowledge/GraphRagKnowledgeRetrievalConfiguration.java
🔇 Additional comments (11)
components/graph-rag-core/src/main/java/com/orgmemory/graphrag/observability/GraphRagEventSink.java (1)
63-71: LGTM!components/graph-rag-core/src/main/java/com/orgmemory/graphrag/observability/FailureTolerantGraphRagEventSink.java (1)
26-57: LGTM!Also applies to: 74-78
components/graph-rag-core/src/test/java/com/orgmemory/graphrag/observability/FailureTolerantGraphRagEventSinkTests.java (1)
21-82: 📐 Maintainability & Code QualityVerify with the JVM test harness.
No terminating clean test result is included for this new sink behavior. Run the relevant Gradle tests to completion;
bootRunis not verification.As per coding guidelines, “Use the testing harness; a terminating clean test is the JVM context gate, and
bootRunis not verification.”Source: Coding guidelines
apps/worker/src/main/java/com/orgmemory/worker/graph/GraphIndexingProcessor.java (1)
82-83: LGTM!core/src/main/java/com/orgmemory/core/knowledge/GraphRagKnowledgeRetrievalConfiguration.java (1)
43-44: LGTM!integrations/graph-rag-observability/src/main/java/com/orgmemory/integrations/graphrag/observability/ExceptionSanitizingSpanExporter.java (1)
41-136: LGTM!integrations/graph-rag-observability/src/test/java/com/orgmemory/integrations/graphrag/observability/ExceptionSanitizingSpanExporterTests.java (1)
35-112: LGTM!integrations/graph-rag-observability/src/main/java/com/orgmemory/integrations/graphrag/observability/SpanExportSanitizationAutoConfiguration.java (1)
18-30: LGTM!integrations/graph-rag-observability/src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports (1)
2-2: LGTM!integrations/graph-rag-observability/src/test/java/com/orgmemory/integrations/graphrag/observability/SpanExportSanitizationAutoConfigurationTests.java (1)
30-86: LGTM!integrations/graph-rag-observability/build.gradle.kts (1)
8-19: 📐 Maintainability & Code QualityRun the module test harness before merging.
#!/bin/bash set -euo pipefail ./gradlew :integrations:graph-rag-observability:test --no-daemonAs per coding guidelines, “Use the testing harness; a terminating clean test is the JVM context gate, and
bootRunis not verification.”
Addresses three review findings, one of which corrects a claim this branch made about its own change. **The YAML pin is a default, not enforcement.** The previous commit described it as "deliberately not environment-overridable". That was wrong: LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_AI_OPENAI, a system property, a logback configuration, or a level on a more specific logger all outrank application.yml. Configuration cannot enforce a boundary configuration can undo. ProviderLoggingBoundaryVerifier therefore checks the resolved state instead of any source of it: each class holding a payload-logging call site is asked whether WARN is enabled on its own logger — the exact question its isWarnEnabled() guard will ask, with inheritance already applied — and startup fails if any answers yes. Failing beats starting, because the alternative is an application that looks healthy while writing customer text to disk. No disable property. **The sink log could still flood.** Reporting on a change of kind meant a backend alternating between two exceptions logged at event rate, which a timeout retrying as a connection failure produces immediately. Reporting is now keyed on the set of kinds seen, bounded at ten, with an A->B->A test. **The profile scan had two holes.** It skipped `.yaml` files, and it matched only ancestors of the guarded packages, so a child such as `org.springframework.ai.openai.api: WARN` under an ERROR parent passed. The YAML comments and test javadoc that repeated the overclaim now say plainly that they cover the shipped default and point at the verifier for enforcement. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 0 of the observability pipeline increment. The runbook has been promising payload-free telemetry; three paths were bypassing it.
What was wrong
Provider libraries log prompts.
spring.ai.chat.observations.log-prompt: falsegoverns Spring AI's observation handlers, not the client libraries, which build their own WARN messages by concatenating the prompt or the response content —OpenAiChatModel:222,AnthropicChatModel:554,:1219, and a fourth found during the sweep,:1018. Production runs at root INFO, so nothing suppressed them.Error spans carry exception text. Micrometer's bridge ends a failed span with
recordException(throwable)andsetStatus(ERROR, throwable.getMessage()). The runbook's release gate — "confirm error spans contain no exception event or stack trace" — could not have passed.A broken sink was invisible. Six call sites caught and discarded emission failures, correctly, so a sink failing since startup looked exactly like a quiet one.
Nothing has leaked: the ZM deployment is a POC with no real users or customer data, so this is preventive.
What changed
Provider packages are pinned above WARN in the base configuration of both apps — a development database holds real uploaded documents too — and deliberately without an environment override. Every leak site is guarded by
isWarnEnabled(), so the pin stops the message being built, not merely printed.ExceptionSanitizingSpanExporterkeepsexception.typealone and clears the status description, as the last gate before egress. Micrometer'sSpanFilterreaches the events but not the description, which is why this sits at theSpanDatalevel. No toggle: a payload guard a deployment can switch off is not a guard.FailureTolerantGraphRagEventSinkkeeps absorbing sink failures but counts them, records the failure type and logs once per change of kind. Class names only — a telemetry error can quote the request it failed to send. UsesSystem.Logger, sograph-rag-corekeeps its zero-runtime-dependency property.Scope this does not cover
Span attributes are not filtered, and no test asserts the whole export against the allowlist. Both are stated in the test matrix rather than glossed; the whole-export test is phase 5.
Also found
Checking LightRAG's tracing against the pinned v1.5.4 checkout corrected an earlier claim in this increment that it has none. It has one integration —
lightrag/llm/openai.pyswaps inlangfuse.openai.AsyncOpenAI, which captures prompts and completions verbatim — for the OpenAI binding only, with no masking anywhere.The comparison surfaced a gap here:
Stagedeclares fourteen values and production emits ten.PARSE,CHUNK,GLEANandGENERATEhave no producer, and deletion/rebuild has no stage at all. Recorded as phase 2 work.Verification
:core:test,:components:graph-rag-core:test,:integrations:graph-rag-observability:test, pluscompileJava compileTestJavaacross the repo — all green locally. Specs and test matrix reconciled in the same change, since their source paths cover the code that moved.🤖 Generated with Claude Code
Summary by CodeRabbit
Bug Fixes
Tests