Skip to content

Reduce tracing volume for read-heavy paths#164

Merged
sjmiller609 merged 3 commits into
mainfrom
codex/trace-volume-control
Mar 25, 2026
Merged

Reduce tracing volume for read-heavy paths#164
sjmiller609 merged 3 commits into
mainfrom
codex/trace-volume-control

Conversation

@sjmiller609

@sjmiller609 sjmiller609 commented Mar 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • sample successful root GET traces at 10% by default via OTEL config
  • keep sampled-out GET errors visible with a compact fallback HTTP span
  • suppress noisy vm info tracing on read-heavy status paths while keeping outer GetVMInfo error visibility

Testing

  • go test ./lib/otel ./lib/hypervisor ./lib/vmm ./cmd/api/config
  • go test ./lib/hypervisor/firecracker -run TestNonexistent -count=0
  • GOOS=darwin GOARCH=arm64 go test ./lib/hypervisor/vz -run TestNonexistent -count=0
  • go test ./cmd/api -run TestNonexistent -count=0

Note

Medium Risk
Changes OpenTelemetry sampling and selectively suppresses hypervisor/VMM HTTP spans, which could hide useful debugging data if misconfigured, but does not affect request handling or persistence paths.

Overview
Reduces OpenTelemetry trace volume by adding configurable sampling for successful server GET requests via otel.successful_get_sample_ratio (default 0.1), wiring it through config/env, validation, initialization, and documentation.

Suppresses tracing for noisy hypervisor status GET calls (notably GET / and GET /api/v1/vm.info) across Firecracker, VZ shim, and Cloud Hypervisor VMM HTTP clients, while adding an error-only detached span for GetVMInfo failures to preserve visibility. Includes targeted unit tests to lock in the new defaults and suppression/sampling behavior.

Written by Cursor Bugbot for commit 0cacf2c. This will update automatically on new commits. Configure here.

@sjmiller609 sjmiller609 requested a review from hiroTamada March 24, 2026 20:54
@sjmiller609 sjmiller609 marked this pull request as ready for review March 24, 2026 20:54
Comment thread lib/otel/http_sampling.go

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Comment thread lib/otel/http_sampling.go
@sjmiller609 sjmiller609 merged commit 0414092 into main Mar 25, 2026
6 checks passed
@sjmiller609 sjmiller609 deleted the codex/trace-volume-control branch March 25, 2026 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants