Skip to content

feat(observability): connect event bus components - #91

Merged
FrankSpitulski merged 10 commits into
mainfrom
codex/observability/event-bus-components
Jul 31, 2026
Merged

feat(observability): connect event bus components#91
FrankSpitulski merged 10 commits into
mainfrom
codex/observability/event-bus-components

Conversation

@FrankSpitulski

@FrankSpitulski FrankSpitulski commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • inject the DSX OpenTelemetry SDK configuration and collector sidecar into Auth Callout pods
  • expose NACK controller-runtime metrics through a Service and ServiceMonitor
  • align Auth Callout, NACK, and Surveyor monitors to the DSX 30s interval / 10s timeout contract
  • order local observability fixtures before Event Bus deployment and add live endpoint, monitor, and injection checks
  • document the observability state of every Event Bus component

Component inventory

Component Logs Metrics Traces
NATS stdout/stderr Surveyor upstream unsupported
NATS mTLS stdout/stderr Surveyor through federated SYS account upstream unsupported
Auth Callout structured stdout ServiceMonitor DSX OTel sidecar
NACK stdout/stderr ServiceMonitor upstream unsupported
Surveyor stdout/stderr ServiceMonitor upstream unsupported

Agent Gateway dataplane, controller, rate-limit service, Valkey exporter, and
bridge were already connected and remain covered by the shared live
observability tests.

Validation

  • make check
  • helm lint auth-callout/deploy
  • helm lint deploy/nats-event-bus
  • clean make test
    • Event Bus functional: pass
    • Event Bus performance: 12/12 pass
    • Agent Gateway functional including new observability checks: 63/63 pass
    • Agent Gateway performance thresholds: pass

Summary by CodeRabbit

  • New Features

    • Added centralized local OpenTelemetry infrastructure for tracing and metrics.
    • Added optional operator-based tracing injection with configurable instrumentation and sidecar references.
    • Added NACK metrics monitoring with configurable labels, scrape intervals, and timeouts.
    • Expanded monitoring coverage for auth-callout and NATS Surveyor metrics.
  • Documentation

    • Expanded deployment and tracing guidance, including direct OTLP export and optional injection.
  • Improvements

    • Added configurable Prometheus scrape timeouts and improved tracing configuration validation.

Signed-off-by: Frank Spitulski <fspitulski@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Jul 31, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f48f7160-8f30-42d2-afcf-bc7322dbcf3b

📥 Commits

Reviewing files that changed from the base of the PR and between ed56e22 and c9cf06e.

📒 Files selected for processing (1)
  • local/agent-gateway/tests/functional/session_protocol_health_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • local/agent-gateway/tests/functional/session_protocol_health_test.go

📝 Walkthrough

Walkthrough

Auth-callout now supports direct OTLP export by default and optional OpenTelemetry Operator injection. NACK metrics monitoring resources and scrape timeouts were added. Local OpenTelemetry deployment wiring, documentation, and functional observability checks were updated.

Changes

Observability deployment and validation

Layer / File(s) Summary
Auth-callout tracing and metrics
auth-callout/deploy/values.yaml, auth-callout/deploy/templates/deployment.yaml, auth-callout/deploy/templates/servicemonitor.yaml, auth-callout/deploy/README.md
Auth-callout exports traces directly by default, supports optional operator injection with configurable references, and uses a 30-second scrape interval with a 10-second timeout.
NACK metrics monitoring
deploy/nats-event-bus/values.yaml, deploy/nats-event-bus/templates/nack-monitoring.yaml
NACK metrics configuration and a conditional PodMonitor expose and scrape controller-runtime metrics with configurable labels, selectors, interval, timeout, and port relabeling.
Local observability infrastructure
local/infra/observability/*, local/agent-gateway/skaffold.fixtures.yaml, local/event-bus/skaffold.yaml, local/event-bus/k8s/local-dev-values.yaml
Local resources and Skaffold wiring deploy the OpenTelemetry operator and collector in the dsx-obs namespace and enable auth-callout injection in local development.
Deployment documentation and validation
deploy/README.md, local/agent-gateway/tests/functional/session_protocol_health_test.go
Documentation describes auth-callout, NACK, and Surveyor observability. Functional tests validate metric endpoints, monitoring resources, selectors, relabeling, and auth-callout OpenTelemetry configuration.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Skaffold
  participant OTelOperator
  participant OTelCollector
  participant AuthCallout
  participant Prometheus
  participant NACK
  Skaffold->>OTelOperator: Deploy operator configuration
  Skaffold->>OTelCollector: Deploy collector configuration
  OTelOperator->>AuthCallout: Inject tracing configuration when enabled
  AuthCallout->>OTelCollector: Export OTLP/gRPC traces
  Prometheus->>AuthCallout: Scrape /metrics with configured timeout
  Prometheus->>NACK: Scrape /metrics with configured timeout
Loading

Suggested reviewers: rapphil

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the observability integration for Event Bus components, which is the primary change in the pull request.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/observability/event-bus-components

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies"


Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
deploy/nats-event-bus/templates/nack-monitoring.yaml (1)

22-23: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Keep the metrics Service selector synchronized with NACK label mode.

The selector is hard-coded to app: nack. The upstream NACK chart supports switching between legacy app labels and modern app.kubernetes.io/* labels; with the modern mode, this Service can have no endpoints. Derive the selector from the pinned chart helper or condition it on the selected label mode, then render both modes with helm lint. (raw.githubusercontent.com)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@deploy/nats-event-bus/templates/nack-monitoring.yaml` around lines 22 - 23,
Update the metrics Service selector in nack-monitoring.yaml to follow the pinned
NACK chart’s selected label mode instead of hard-coding app: nack. Reuse the
chart helper or conditionally render the corresponding legacy app labels and
modern app.kubernetes.io labels, and verify both modes with helm lint.

Source: MCP tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@auth-callout/deploy/values.yaml`:
- Line 408: Align the Auth Callout scrape contract by changing the default
interval to 30s in auth-callout/deploy/values.yaml at lines 408-408 and updating
the example interval to 30s in auth-callout/deploy/README.md at lines 280-280;
keep the 10s scrape timeout unchanged.

In `@skaffold.yaml`:
- Around line 9-11: Update the event-bus configuration in skaffold.yaml to add
../agent-gateway/skaffold.fixtures.yaml with config agent-gateway-fixtures to
its requires list, ensuring the agent-gateway fixtures deploy before event-bus.

---

Nitpick comments:
In `@deploy/nats-event-bus/templates/nack-monitoring.yaml`:
- Around line 22-23: Update the metrics Service selector in nack-monitoring.yaml
to follow the pinned NACK chart’s selected label mode instead of hard-coding
app: nack. Reuse the chart helper or conditionally render the corresponding
legacy app labels and modern app.kubernetes.io labels, and verify both modes
with helm lint.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1f05ea32-7d78-4b91-8bea-58091821def3

📥 Commits

Reviewing files that changed from the base of the PR and between c8fb031 and 4804a63.

📒 Files selected for processing (10)
  • auth-callout/deploy/README.md
  • auth-callout/deploy/templates/deployment.yaml
  • auth-callout/deploy/templates/servicemonitor.yaml
  • auth-callout/deploy/values.yaml
  • deploy/README.md
  • deploy/nats-event-bus/templates/nack-monitoring.yaml
  • deploy/nats-event-bus/values.yaml
  • local/agent-gateway/tests/functional/session_protocol_health_test.go
  • local/event-bus/k8s/local-dev-values.yaml
  • skaffold.yaml
💤 Files with no reviewable changes (1)
  • local/event-bus/k8s/local-dev-values.yaml

Comment thread auth-callout/deploy/values.yaml
Comment thread skaffold.yaml Outdated
Signed-off-by: Frank Spitulski <fspitulski@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@auth-callout/deploy/values.yaml`:
- Around line 258-265: Update the default observability tracing configuration in
values.yaml so tracing is disabled when operatorInjection.enabled is false, or
configure an explicit non-sidecar exporter endpoint. Keep the default settings
consistent so a chart install does not target the unavailable 127.0.0.1:4317
collector.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0cde1490-c435-4750-953a-ee48966b78a8

📥 Commits

Reviewing files that changed from the base of the PR and between 4804a63 and c2fbaa1.

📒 Files selected for processing (5)
  • auth-callout/deploy/README.md
  • auth-callout/deploy/templates/deployment.yaml
  • auth-callout/deploy/values.yaml
  • deploy/nats-event-bus/values.yaml
  • local/event-bus/skaffold.yaml
🚧 Files skipped from review as they are similar to previous changes (1)
  • auth-callout/deploy/README.md

Comment thread auth-callout/deploy/values.yaml
Signed-off-by: Frank Spitulski <fspitulski@nvidia.com>
Signed-off-by: Frank Spitulski <fspitulski@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@auth-callout/deploy/values.yaml`:
- Around line 262-264: Update the OpenTelemetry configuration around
instrumentationRef and sidecarRef so direct Helm deployments either document and
provision the required dsx-obs/default-instrumentation and
dsx-obs/default-sidecar resources first, or disable injection when those
resources are unavailable; preserve the current enabled behavior when the
prerequisites exist.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 708e1825-86d2-4cba-83c3-83f78fdc991f

📥 Commits

Reviewing files that changed from the base of the PR and between 2aba5b1 and 45a9206.

📒 Files selected for processing (3)
  • auth-callout/deploy/README.md
  • auth-callout/deploy/values.yaml
  • deploy/nats-event-bus/values.yaml
💤 Files with no reviewable changes (1)
  • deploy/nats-event-bus/values.yaml

Comment thread auth-callout/deploy/values.yaml Outdated
Signed-off-by: Frank Spitulski <fspitulski@nvidia.com>
Signed-off-by: Frank Spitulski <fspitulski@nvidia.com>
Signed-off-by: Frank Spitulski <fspitulski@nvidia.com>
Signed-off-by: Frank Spitulski <fspitulski@nvidia.com>
@FrankSpitulski
FrankSpitulski marked this pull request as ready for review July 31, 2026 03:24
@FrankSpitulski
FrankSpitulski requested a review from a team July 31, 2026 03:24
@github-actions

Copy link
Copy Markdown

🔐 TruffleHog Secret Scan

No secrets or credentials found!

Your code has been scanned for 700+ types of secrets and credentials. All clear! 🎉

🔗 View scan details

🕐 Last updated: 2026-07-31 03:25:29 UTC | Commit: 7705d98

@github-actions

Copy link
Copy Markdown

🛡️ CodeQL Analysis

🚨 Found 1 issue(s)

Severity Breakdown:

  • 🔴 Errors: 0
  • 🟡 Warnings: 0
  • 🔵 Notes: 0
📋 Top Issues

💡 Note: Enable GitHub Advanced Security to see full details in the Security tab.

🕐 Last updated: 2026-07-31 03:27:12 UTC | Commit: 7705d98

Comment thread deploy/nats-event-bus/templates/nack-monitoring.yaml Outdated
Signed-off-by: Frank Spitulski <fspitulski@nvidia.com>
@github-actions

Copy link
Copy Markdown

@FrankSpitulski
FrankSpitulski enabled auto-merge (squash) July 31, 2026 19:33

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
local/agent-gateway/tests/functional/session_protocol_health_test.go (1)

345-345: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Gate the auth-callout case on operator injection.

Skip the case unless auth-callout.observability.tracing.operatorInjection.enabled is true. The chart omits the injection annotations and sidecar when this value is false, so the unconditional assertions fail in valid direct-export deployments.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@local/agent-gateway/tests/functional/session_protocol_health_test.go` at line
345, Update the “auth callout” case in the session protocol health tests to run
only when auth-callout operator injection is enabled via
auth-callout.observability.tracing.operatorInjection.enabled. Preserve the
existing assertions for injected deployments while skipping this case for
direct-export configurations where injection is disabled.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@local/agent-gateway/tests/functional/session_protocol_health_test.go`:
- Around line 269-279: Extend the relabeling validation in the nack branch of
the session protocol health test to inspect relabeling["sourceLabels"] and
assert it contains exactly "__meta_kubernetes_pod_ip", alongside the existing
targetLabel and replacement checks.
- Around line 244-248: Update the pod selector validation in the podMonitor
branch to pass a status.phase=Running filter to runner.ListPods. Keep the
existing empty-match failure behavior and diagnostic message, while ensuring
completed or failed Pods do not satisfy the selector check.
- Around line 190-193: Update the NACK metrics probe around firstRunningPodName
to retrieve the rendered PodMonitor/nack spec.selector.matchLabels and pass that
complete label set to firstRunningPodName instead of hardcoding app=nack.
Preserve the existing metrics endpoint validation after selecting the pod.

---

Outside diff comments:
In `@local/agent-gateway/tests/functional/session_protocol_health_test.go`:
- Line 345: Update the “auth callout” case in the session protocol health tests
to run only when auth-callout operator injection is enabled via
auth-callout.observability.tracing.operatorInjection.enabled. Preserve the
existing assertions for injected deployments while skipping this case for
direct-export configurations where injection is disabled.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 18b91be9-37a4-4aa7-a62e-23ead56803f0

📥 Commits

Reviewing files that changed from the base of the PR and between 7705d98 and ed56e22.

📒 Files selected for processing (8)
  • deploy/README.md
  • deploy/nats-event-bus/templates/nack-monitoring.yaml
  • deploy/nats-event-bus/values.yaml
  • local/agent-gateway/skaffold.fixtures.yaml
  • local/agent-gateway/skaffold.infra.yaml
  • local/agent-gateway/tests/functional/session_protocol_health_test.go
  • local/event-bus/k8s/local-dev-values.yaml
  • local/event-bus/skaffold.yaml
🚧 Files skipped from review as they are similar to previous changes (6)
  • local/event-bus/skaffold.yaml
  • local/agent-gateway/skaffold.fixtures.yaml
  • local/agent-gateway/skaffold.infra.yaml
  • deploy/nats-event-bus/values.yaml
  • deploy/README.md
  • local/event-bus/k8s/local-dev-values.yaml

Comment thread local/agent-gateway/tests/functional/session_protocol_health_test.go Outdated
Signed-off-by: Frank Spitulski <fspitulski@nvidia.com>
@FrankSpitulski
FrankSpitulski disabled auto-merge July 31, 2026 19:38
@FrankSpitulski
FrankSpitulski enabled auto-merge (squash) July 31, 2026 19:49
@FrankSpitulski
FrankSpitulski merged commit 1f5526b into main Jul 31, 2026
22 checks passed
@FrankSpitulski
FrankSpitulski deleted the codex/observability/event-bus-components branch July 31, 2026 20:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants