Objective
Add a ready-to-use KQL query section to docs/telemetry.md so users who send eval traces to Azure Monitor / Application Insights can immediately query their data.
Background
When eval traces land in App Insights (via OTLP ingestion — see #88), all span attributes are available as custom dimensions in the dependencies and requests tables. Users need practical KQL examples to extract value from this data.
Proposed Queries
Add a new section between 'Sending Traces to Azure Monitor' and 'Evaluation Tracing vs. Agent Execution Tracing' with queries like:
- Slowest evaluation rows — top N
eval_item spans by duration
- Failed evaluators — filter by
agentops.eval.evaluator.passed == false
- Pass rate over time — trend of
agentops.eval.pass_rate from root spans
- Token usage per run — sum of
gen_ai.usage.input_tokens + output_tokens
- Evaluator score distribution — histogram of
agentops.eval.evaluator.score grouped by evaluator name
Format
Each query should include:
- A short description of what it answers
- The KQL query itself (ready to paste into App Insights → Logs)
- Expected table/column names based on OTLP → App Insights mapping
Acceptance Criteria
Dependencies
Objective
Add a ready-to-use KQL query section to
docs/telemetry.mdso users who send eval traces to Azure Monitor / Application Insights can immediately query their data.Background
When eval traces land in App Insights (via OTLP ingestion — see #88), all span attributes are available as custom dimensions in the
dependenciesandrequeststables. Users need practical KQL examples to extract value from this data.Proposed Queries
Add a new section between 'Sending Traces to Azure Monitor' and 'Evaluation Tracing vs. Agent Execution Tracing' with queries like:
eval_itemspans by durationagentops.eval.evaluator.passed == falseagentops.eval.pass_ratefrom root spansgen_ai.usage.input_tokens+output_tokensagentops.eval.evaluator.scoregrouped by evaluator nameFormat
Each query should include:
Acceptance Criteria
docs/telemetry.mddependencies,requests,customDimensions)Dependencies