test: improve tracing span attribution and add span regression tests#264
Merged
Conversation
The runner emitted no parent span for an agent loop run, so tool spans landed as orphan siblings in Tempo: a single failing task could not be recovered as one trace, only as a fan of unrelated spans. Worse, four call sites passed `task_id` into the helpers' `operation_id` parameter (runner.rs:311 et al.), so the `attack_operation_id` field on tool spans actually held a task ID — silently breaking any dashboard that filtered by operation. - Open `agent.loop` once at the top of run_agent_loop, carrying op.id, task.id, agent.role, agent.model. Move the existing body into run_agent_loop_inner and instrument the future with that span; every child span now inherits the parent. - Add `task_id` to AgentSpanBuilder + the four trace_* helpers and emit `op.id` / `task.id` alongside the existing `attack_operation_id` field for back-compat with current dashboards. - Fix the four runner.rs sites that were passing task_id where operation_id was expected, and pass real op.id and task.id to the worker discovery span and the orchestrator domain-admin span. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…tokens Without per-call attribution there was no way to tell whether a slow agent loop was burning time inside provider.chat (network/LLM) or between calls (tool dispatch, queue waits). Token spend was visible only on the session log, not in Tempo traces. Open one `llm.call` span per retry attempt around `provider.chat`. After the call returns, record duration_ms plus the four token-usage counters and stop_reason as span attributes; on error, record the formatted error. Each retry gets its own span so a 429 backoff does not inflate the duration attributed to the eventual successful call. The span also carries `task.id`, `llm.model`, `llm.attempt`, and the request shape (tool/message counts) so Tempo queries can isolate slow calls without joining other spans. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
… and task builders **Added:** - Added `tracing::info_span` and `.instrument()` to automation task spawning for correlating spans by `automation.kind` in `automation_spawner.rs` - Added `tracing::info_span` to `Dispatcher::throttled_submit` and propagated span context to inner calls for dispatch, throttle, and defer decisions in `submission.rs` - Added detailed span field recording for throttle decisions and task IDs in `submission.rs` - Added `#[instrument]` attribute with relevant fields to all public task request methods in `task_builders.rs` to improve observability of automation requests **Changed:** - Refactored task submission logic in `Dispatcher` to utilize structured spans, enabling improved tracing and debugging of automation task lifecycles - Updated imports to include necessary `tracing` macros and types in affected modules
**Added:** - Added `tracing-test` and `tracing-subscriber` as dev dependencies in `ares-llm/Cargo.toml` and updated `Cargo.lock` - Introduced `ares-llm/tests/common/mod.rs` and `span_capture.rs` providing a test-only tracing layer and helpers to capture and inspect spans for integration testing - Added `ares-llm/tests/span_regressions.rs` with regression tests to verify the presence and correctness of tracing spans and their attributes in the agent loop, covering op/task ID separation, token usage, error reporting, and retry attempt span emission **Changed:** - Updated `windows-sys` dependency from 0.48.0 to 0.61.2 in `Cargo.lock`
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #264 +/- ##
==========================================
+ Coverage 75.11% 75.29% +0.17%
==========================================
Files 383 384 +1
Lines 81492 81639 +147
==========================================
+ Hits 61215 61470 +255
+ Misses 20277 20169 -108
🚀 New features to boost your workflow:
|
**Changed:** - Reformatted the `info_span!` macro call in `spawn_automation_tasks` for improved code consistency and readability by moving parameters onto a single line
…and agent loop **Changed:** - Simplified the creation of `info_span!` macro invocation by removing unnecessary newlines and formatting in `automation_spawner.rs` - Updated several calls in `runner.rs` to pass `op_id` directly instead of as a reference, streamlining argument types and reducing unnecessary references - Reformatted line wrapping in `span_capture.rs` for improved readability without changing logic
**Added:** - Introduced `codecov.yml` to configure coverage thresholds, status checks, and comment behavior for automated code coverage reporting in CI
…mentation' into feat/telemetry-tracing-test-regressions
l50
added a commit
that referenced
this pull request
May 9, 2026
…264) **Key Changes:** - Enhanced tracing span attribution with proper `op.id` and `task.id` separation - Instrumented automation and dispatcher task submission with correlated spans - Added comprehensive regression tests to verify span emission and attributes - Introduced `tracing-test` and custom span capture utilities for testing **Added:** - Regression tests for span emission and correctness in `ares-llm/tests/span_regressions.rs` - Common test helpers and a custom `SpanCapture` tracing subscriber in `ares-llm/tests/common/` - `tracing-test` and `tracing-subscriber` dependencies for enhanced test coverage and span capture **Changed:** - Instrumented all automation dispatcher task builder methods in `ares-cli` with `#[instrument]` to emit detailed spans for each automation action - Updated `throttled_submit` and submission flow in `ares-cli` to record decision and task IDs in spans, ensuring correlation across async boundaries - Modified `spawn_automation_tasks` to wrap spawned automation tasks in their own root spans with `automation.kind` for better trace correlation in Tempo - Enhanced telemetry span builders in `ares-core` to support both `op.id` and `task.id` attributes and propagate them throughout the operation and agent task lifecycle - Refactored agent loop in `ares-llm` to emit a single parent span per agent task, with all child spans inheriting correct operation and task context - Updated all internal tracing helpers in `ares-core` to accept and propagate both `operation_id` and `task_id` - Bumped `actions/upload-artifact` and `actions/setup-go` versions in GitHub Actions workflows for security and compatibility - Updated several dependencies in `Cargo.lock`, including `windows-sys`, and added tracing-related crates for test instrumentation **Removed:** - Orphaned parent context for spawned automation tasks—now every automation task and dispatcher submission is properly correlated in tracing backends ---------
l50
added a commit
that referenced
this pull request
May 9, 2026
…264) **Key Changes:** - Enhanced tracing span attribution with proper `op.id` and `task.id` separation - Instrumented automation and dispatcher task submission with correlated spans - Added comprehensive regression tests to verify span emission and attributes - Introduced `tracing-test` and custom span capture utilities for testing **Added:** - Regression tests for span emission and correctness in `ares-llm/tests/span_regressions.rs` - Common test helpers and a custom `SpanCapture` tracing subscriber in `ares-llm/tests/common/` - `tracing-test` and `tracing-subscriber` dependencies for enhanced test coverage and span capture **Changed:** - Instrumented all automation dispatcher task builder methods in `ares-cli` with `#[instrument]` to emit detailed spans for each automation action - Updated `throttled_submit` and submission flow in `ares-cli` to record decision and task IDs in spans, ensuring correlation across async boundaries - Modified `spawn_automation_tasks` to wrap spawned automation tasks in their own root spans with `automation.kind` for better trace correlation in Tempo - Enhanced telemetry span builders in `ares-core` to support both `op.id` and `task.id` attributes and propagate them throughout the operation and agent task lifecycle - Refactored agent loop in `ares-llm` to emit a single parent span per agent task, with all child spans inheriting correct operation and task context - Updated all internal tracing helpers in `ares-core` to accept and propagate both `operation_id` and `task_id` - Bumped `actions/upload-artifact` and `actions/setup-go` versions in GitHub Actions workflows for security and compatibility - Updated several dependencies in `Cargo.lock`, including `windows-sys`, and added tracing-related crates for test instrumentation **Removed:** - Orphaned parent context for spawned automation tasks—now every automation task and dispatcher submission is properly correlated in tracing backends ---------
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Key Changes:
op.idandtask.idseparationtracing-testand custom span capture utilities for testingAdded:
ares-llm/tests/span_regressions.rsSpanCapturetracing subscriber inares-llm/tests/common/tracing-testandtracing-subscriberdependencies for enhanced test coverage and span captureChanged:
ares-cliwith#[instrument]to emit detailed spans for each automation actionthrottled_submitand submission flow inares-clito record decision and task IDs in spans, ensuring correlation across async boundariesspawn_automation_tasksto wrap spawned automation tasks in their own root spans withautomation.kindfor better trace correlation in Tempoares-coreto support bothop.idandtask.idattributes and propagate them throughout the operation and agent task lifecycleares-llmto emit a single parent span per agent task, with all child spans inheriting correct operation and task contextares-coreto accept and propagate bothoperation_idandtask_idactions/upload-artifactandactions/setup-goversions in GitHub Actions workflows for security and compatibilityCargo.lock, includingwindows-sys, and added tracing-related crates for test instrumentationRemoved: