fix(ai-sdk): agent name in span names when available #838
Conversation
WalkthroughAdds top-level AI span renaming to use agent metadata, new AI span type constants (generateObject/streamText/streamObject), cache-related token attribute transformations, expanded semantic attributes, and test/fixture additions for agent-aware span naming and cache-token handling. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes
Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Comment |
… into gk/ai-sdk-fixes
- Keep original span names (ai.generateText, ai.streamText, etc.) when no agent metadata - Only transform top-level spans to agent name when agent metadata is present - Transform child spans (text.generate, object.generate) to OpenLLMetry format - Updated unit test to pass agent name instead of deprecated "run.ai" name - Added tests for generateObject and streamText with agent metadata - Updated test recordings to match new behavior Fixes customer complaint where generic "AI" or "run" names appeared instead of actual agent names 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed everything up to ec097de in 2 minutes and 25 seconds. Click for details.
- Reviewed
965lines of code in7files - Skipped
0files when reviewing. - Skipped posting
1draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:515
- Draft comment:
Bug: In transformAiSdkSpanNames the agent name is retrieved using the key constructed from AI_TELEMETRY_METADATA_PREFIX ('ai.telemetry.metadata.agent'), but transformTelemetryMetadata deletes these keys and instead sets the agent name on the GEN_AI_AGENT_NAME attribute. Use span.attributes[SpanAttributes.GEN_AI_AGENT_NAME] instead. - Reason this comment was not posted:
Decided after close inspection that this draft comment was likely wrong and/or not actionable: usefulness confidence = 20% vs. threshold = 50% The comment is making an assumption about execution order that I cannot verify from this file alone. The two functions are separate exports -transformAiSdkSpanNamesandtransformAiSdkSpanAttributes. Without seeing where these are called in the broader codebase, I cannot determine iftransformTelemetryMetadata(which deletes the metadata prefix keys and sets GEN_AI_AGENT_NAME) runs before or aftertransformAiSdkSpanNames. The comment could be correct, but it requires cross-file context to verify. According to the rules, I should delete comments that require more context to understand. I'm assuming the comment is wrong because I can't verify the execution order, but the reviewer might have knowledge of the calling code that showstransformTelemetryMetadatadoes run first. The comment provides a specific, actionable fix which suggests the reviewer has investigated this. Even if the reviewer has investigated this, the rule states I need STRONG EVIDENCE from the diff itself. The execution order is not clear from this file alone, and understanding this requires seeing other files or parts of the codebase. This falls under "cross-file issues" which I should ignore. Delete this comment because verifying whether it's correct requires understanding the execution order of these functions across multiple files, which is cross-file context I don't have access to in this diff.
Workflow ID: wflow_PbFsV69q7gHjvGZo
You can customize 
Changed test name from "should use default 'run.ai' span name" to "should preserve original AI SDK span name" to accurately reflect that we now preserve the original AI SDK span names (ai.generateText, etc.) when no agent metadata is provided, rather than transforming to "run.ai" Created new test recording to match updated test name 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (1)
430-443: Deduplicate the list of top-level AI span namesYou now maintain two separate arrays of top-level AI span identifiers:
- An inline
topLevelSpanNamesinsidetransformTelemetryMetadata(Lines 430-438).- The module-level
TOP_LEVEL_AI_SPANS(Lines 501-507).Keeping these in sync is error-prone when adding new top-level span types. Consider reusing
TOP_LEVEL_AI_SPANSintransformTelemetryMetadata(or centralizing the list elsewhere) to avoid future drift.Also applies to: 501-507
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (7)
packages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-propagate-agent-name-to-tool-call-spans_3577231859/recording.har(10 hunks)packages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-use-agent-name-for-generateObject-with-agent-metadata_1744675110/recording.har(1 hunks)packages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-use-agent-name-for-streamText-with-agent-metadata_4019571713/recording.har(1 hunks)packages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-use-default-run-ai-span-name-when-no-agent-metadata-is-provided_1300307112/recording.har(1 hunks)packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts(3 hunks)packages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts(4 hunks)packages/traceloop-sdk/test/ai-sdk-transformations.test.ts(1 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
packages/{instrumentation-*,traceloop-sdk}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Import AI/LLM semantic attribute constants from @traceloop/ai-semantic-conventions rather than hardcoding strings
Files:
packages/traceloop-sdk/test/ai-sdk-transformations.test.tspackages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.tspackages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts
packages/traceloop-sdk/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
packages/traceloop-sdk/**/*.{ts,tsx}: Use the provided decorators (@workflow, @task, @agent) for workflow/task/agent spans instead of re-implementing them
For manual LLM operations, use trace.withLLMSpan from @traceloop/node-server-sdk
Files:
packages/traceloop-sdk/test/ai-sdk-transformations.test.tspackages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.tspackages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts
**/recordings/**
📄 CodeRabbit inference engine (CLAUDE.md)
Store HTTP interaction recordings for tests under recordings/ directories for Polly.js replay
Files:
packages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-propagate-agent-name-to-tool-call-spans_3577231859/recording.harpackages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-use-agent-name-for-generateObject-with-agent-metadata_1744675110/recording.harpackages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-use-default-run-ai-span-name-when-no-agent-metadata-is-provided_1300307112/recording.harpackages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-use-agent-name-for-streamText-with-agent-metadata_4019571713/recording.har
🧠 Learnings (7)
📓 Common learnings
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/{instrumentation-*,traceloop-sdk}/**/*.{ts,tsx} : Import AI/LLM semantic attribute constants from traceloop/ai-semantic-conventions rather than hardcoding strings
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/instrumentation-*/**/*.{ts,tsx} : Instrumentations must create spans with appropriate AI/LLM semantic attributes for calls they wrap
📚 Learning: 2025-08-24T22:08:07.023Z
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/{instrumentation-*,traceloop-sdk}/**/*.{ts,tsx} : Import AI/LLM semantic attribute constants from traceloop/ai-semantic-conventions rather than hardcoding strings
Applied to files:
packages/traceloop-sdk/test/ai-sdk-transformations.test.tspackages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-use-default-run-ai-span-name-when-no-agent-metadata-is-provided_1300307112/recording.harpackages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.tspackages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts
📚 Learning: 2025-08-24T22:08:07.023Z
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/instrumentation-*/**/*.{ts,tsx} : Instrumentations must create spans with appropriate AI/LLM semantic attributes for calls they wrap
Applied to files:
packages/traceloop-sdk/test/ai-sdk-transformations.test.tspackages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.tspackages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts
📚 Learning: 2025-08-24T22:08:07.023Z
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/traceloop-sdk/**/*.{ts,tsx} : For manual LLM operations, use trace.withLLMSpan from traceloop/node-server-sdk
Applied to files:
packages/traceloop-sdk/test/ai-sdk-transformations.test.tspackages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.tspackages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts
📚 Learning: 2025-08-24T22:08:07.023Z
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/ai-semantic-conventions/src/SemanticAttributes.ts : Define all AI/LLM span attribute constants in packages/ai-semantic-conventions/src/SemanticAttributes.ts
Applied to files:
packages/traceloop-sdk/test/ai-sdk-transformations.test.tspackages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.tspackages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts
📚 Learning: 2025-08-24T22:08:07.023Z
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/traceloop-sdk/**/*.{ts,tsx} : Use the provided decorators (workflow, task, agent) for workflow/task/agent spans instead of re-implementing them
Applied to files:
packages/traceloop-sdk/test/ai-sdk-transformations.test.tspackages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.tspackages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts
📚 Learning: 2025-08-12T13:58:25.583Z
Learnt from: galzilber
Repo: traceloop/openllmetry-js PR: 643
File: packages/traceloop-sdk/src/lib/client/dataset/column.ts:41-47
Timestamp: 2025-08-12T13:58:25.583Z
Learning: In packages/traceloop-sdk/src/lib/client/dataset/column.ts, when the user mentioned "its working there is transform func", they were confirming that the response transformer is functioning correctly and converting snake_case API responses to camelCase, which means the Column class should access camelCase properties (createdAt, updatedAt) from the transformed ColumnResponse data.
Applied to files:
packages/traceloop-sdk/test/ai-sdk-transformations.test.ts
🧬 Code graph analysis (2)
packages/traceloop-sdk/test/ai-sdk-transformations.test.ts (1)
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (1)
transformLLMSpans(459-473)
packages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts (2)
packages/traceloop-sdk/src/lib/tracing/decorators.ts (1)
tool(289-295)packages/ai-semantic-conventions/src/SemanticAttributes.ts (1)
SpanAttributes(23-75)
🔇 Additional comments (7)
packages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-use-default-run-ai-span-name-when-no-agent-metadata-is-provided_1300307112/recording.har (1)
1-172: Test fixture properly structured and placed.The HAR recording is correctly formatted for Polly.js replay, captures a realistic OpenAI API interaction with function-calling tools, and aligns with the test scenario ("no agent metadata, default run.ai span name"). Placement in the recordings/ directory follows coding guidelines.
packages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-use-agent-name-for-streamText-with-agent-metadata_4019571713/recording.har (1)
1-144: HAR fixture structure looks correctRecording follows the existing Polly HAR structure and matches the expected OpenAI streaming response shape; fine as a test fixture.
packages/traceloop-sdk/test/ai-sdk-transformations.test.ts (1)
1777-1780: Root-span simulation now matches production behaviorPassing
"research_assistant"as spanName intotransformLLMSpanscorrectly mirrors the post-name-transformation root span and validates the new agent-root logic.packages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-propagate-agent-name-to-tool-call-spans_3577231859/recording.har (1)
23-167: Updated HAR recording is structurally consistentOnly identifiers, cookie values, and timing/header fields changed; JSON structure is intact and suitable for Polly replay.
packages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-use-agent-name-for-generateObject-with-agent-metadata_1744675110/recording.har (1)
1-172: New generateObject HAR fixture is well-formedRecording cleanly captures the json_schema-based generateObject call and response; appropriate as a Polly fixture.
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (1)
7-24: Agent-aware span naming for top-level vs child AI spans looks correctThe extended constants and updated
transformAiSdkSpanNamescleanly separate concerns:
- Top-level AI spans (
ai.generateText,ai.streamText,ai.generateObject,ai.streamObject) are renamed to the agent name whenai.telemetry.metadata.agentis present, leaving them asai.*when no agent metadata is provided.- Child spans (
*.doGenerate/*.doStream) still map to internal names liketext.generate,object.generate,text.stream,object.stream, so downstream processing remains stable.- Combined with passing
span.nameintotransformLLMSpans, this gives you root spans named after the agent while preserving consistent child span naming.The behavior aligns with the new tests and the stated PR objective.
Also applies to: 501-527
packages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts (1)
20-21: Tests correctly cover agent naming for generateText/generateObject/streamTextThe three tests accurately exercise:
- Default root naming (
"ai.generateText") when no agent metadata is provided- Agent-named root spans (with AGENT span kind and entity name) for
generateObjectandstreamTextwhen agent metadata is present- Proper export flushing via
traceloop.forceFlush()before assertionsThe ai package dependency (
^5.0.22) is properly configured and supportsgenerateObject,streamText, and theexperimental_telemetryoptions used throughout.
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed 3bc47c5 in 44 seconds. Click for details.
- Reviewed
119lines of code in2files - Skipped
0files when reviewing. - Skipped posting
2draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. packages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-preserve-original-AI-SDK-span-name-when-no-agent-metadata-is-provided_1735519430/recording.har:3
- Draft comment:
Updated the recording name and associated timestamps/cookie values to reflect the new behavior, i.e. preserving the original AI SDK span name when no agent metadata is provided. Ensure these fixture updates stay in sync with expected behavior. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
2. packages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts:236
- Draft comment:
Renamed test description from 'use default "run.ai" span name' to 'preserve original AI SDK span name' to better reflect the intended behavior when agent metadata is missing. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
Workflow ID: wflow_rf6T22D6rzzwLjcW
You can customize
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
packages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-preserve-original-AI-SDK-span-name-when-no-agent-metadata-is-provided_1735519430/recording.har (1)
1-172: HAR fixture is well-formed and correctly placed; consider optional sanitization of opaque IDsThe HAR looks structurally valid for Polly.js (HAR 1.2, single POST entry, realistic OpenAI request/response) and the path under
recordings/matches the intended usage for HTTP interaction fixtures. It also correctly represents a “no agent metadata” scenario that your tests rely on.If you want to further harden/simplify fixtures, you could optionally normalize or redact opaque values like
cf-ray,x-request-id,openai-project,_cfuvid, and similar IDs/cookies to reduce churn on re-recording and avoid storing third-party identifiers verbatim in the repo. Not strictly necessary, but worth considering for future recordings.As per coding guidelines, this aligns with the requirement that
**/recordings/**store Polly.js HTTP interaction recordings.packages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts (1)
239-318: Good fallback test for no-agent case, with minor tool duplicationThis test nicely covers the case where AI SDK spans should retain the original
ai.generateTextname and lack agent-related attributes when noagentmetadata is supplied. The only nit is that the calculatortooldefinition duplicates the one in the first test; consider extracting a shared helper to reduce duplication and keep future changes to the tool logic in one place.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (2)
packages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-preserve-original-AI-SDK-span-name-when-no-agent-metadata-is-provided_1735519430/recording.har(1 hunks)packages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts(4 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
**/recordings/**
📄 CodeRabbit inference engine (CLAUDE.md)
Store HTTP interaction recordings for tests under recordings/ directories for Polly.js replay
Files:
packages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-preserve-original-AI-SDK-span-name-when-no-agent-metadata-is-provided_1735519430/recording.har
packages/{instrumentation-*,traceloop-sdk}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Import AI/LLM semantic attribute constants from @traceloop/ai-semantic-conventions rather than hardcoding strings
Files:
packages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts
packages/traceloop-sdk/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
packages/traceloop-sdk/**/*.{ts,tsx}: Use the provided decorators (@workflow, @task, @agent) for workflow/task/agent spans instead of re-implementing them
For manual LLM operations, use trace.withLLMSpan from @traceloop/node-server-sdk
Files:
packages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts
🧠 Learnings (7)
📓 Common learnings
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/instrumentation-*/**/*.{ts,tsx} : Instrumentations must create spans with appropriate AI/LLM semantic attributes for calls they wrap
📚 Learning: 2025-08-24T22:08:07.023Z
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to **/recordings/** : Store HTTP interaction recordings for tests under recordings/ directories for Polly.js replay
Applied to files:
packages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-preserve-original-AI-SDK-span-name-when-no-agent-metadata-is-provided_1735519430/recording.har
📚 Learning: 2025-08-24T22:08:07.023Z
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/{instrumentation-*,traceloop-sdk}/**/*.{ts,tsx} : Import AI/LLM semantic attribute constants from traceloop/ai-semantic-conventions rather than hardcoding strings
Applied to files:
packages/traceloop-sdk/recordings/Test-AI-SDK-Agent-Integration-with-Recording_2039949225/should-preserve-original-AI-SDK-span-name-when-no-agent-metadata-is-provided_1735519430/recording.harpackages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts
📚 Learning: 2025-08-24T22:08:07.023Z
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/traceloop-sdk/**/*.{ts,tsx} : Use the provided decorators (workflow, task, agent) for workflow/task/agent spans instead of re-implementing them
Applied to files:
packages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts
📚 Learning: 2025-08-24T22:08:07.023Z
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/instrumentation-*/**/*.{ts,tsx} : Instrumentations must create spans with appropriate AI/LLM semantic attributes for calls they wrap
Applied to files:
packages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts
📚 Learning: 2025-08-24T22:08:07.023Z
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/traceloop-sdk/**/*.{ts,tsx} : For manual LLM operations, use trace.withLLMSpan from traceloop/node-server-sdk
Applied to files:
packages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts
📚 Learning: 2025-08-24T22:08:07.023Z
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/ai-semantic-conventions/src/SemanticAttributes.ts : Define all AI/LLM span attribute constants in packages/ai-semantic-conventions/src/SemanticAttributes.ts
Applied to files:
packages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts
🧬 Code graph analysis (1)
packages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts (2)
packages/traceloop-sdk/src/lib/tracing/decorators.ts (1)
tool(289-295)packages/ai-semantic-conventions/src/SemanticAttributes.ts (1)
SpanAttributes(23-75)
🔇 Additional comments (4)
packages/traceloop-sdk/test/ai-sdk-agent-integration.test.ts (4)
20-20: Expandedaiimports correctly match new usagesImporting
generateObject,streamText, andtoolalongsidegenerateTextlines up with how the tests use these helpers later in the file; no issues here.
144-147: Root span selection now validates agent-based namingFinding the root span by the agent name and asserting its existence with a clear message directly exercises the new naming behavior for top-level AI spans when agent metadata is present; this is an appropriate and focused expectation.
Also applies to: 158-161
320-378: Solid coverage forgenerateObjectagent naming and attributesThis test cleanly verifies that
generateObjectspans are renamed to the agent name and carry the expectedGEN_AI_AGENT_NAME,TRACELOOP_SPAN_KIND = "agent", andTRACELOOP_ENTITY_NAMEattributes when agent metadata is present, which aligns with the intended transformation behavior and usesSpanAttributesconstants instead of raw strings, as per semantic-conventions guidance.
380-437: Streaming scenario is well-covered for agent-based span namingThe
streamTexttest fully consumes the stream and then asserts that the root span is renamed to the agent and tagged with the correct agent-related attributes, giving good coverage of the streaming path for the new naming logic.
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed 22b79f9 in 1 minute and 19 seconds. Click for details.
- Reviewed
1136lines of code in6files - Skipped
0files when reviewing. - Skipped posting
8draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:35
- Draft comment:
New constants for cache tokens are defined. Consider adding explicit type annotations for clarity. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
2. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:374
- Draft comment:
transformCacheCreationInputTokens correctly maps and removes the legacy attribute. Consider adding a comment on expected value type (e.g. number or string). - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
3. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:383
- Draft comment:
transformCacheReadInputTokens implementation is clear and follows the pattern. No issues detected. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
4. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:392
- Draft comment:
transformCachedInputTokens properly overwrites the cache read attribute if present; the ordering of these functions is critical. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
5. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:501
- Draft comment:
The order of cache token transformation calls (creation, read, then cached) ensures that 'cachedInputTokens' takes precedence. Consider adding an inline comment explaining this ordering for future maintainability. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
6. packages/traceloop-sdk/test/ai-sdk-integration.test.ts:273
- Draft comment:
New integration test for cache tokens transformation is comprehensive; it verifies removal of old keys and correct mapping of cache creation and read tokens. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
7. packages/traceloop-sdk/test/ai-sdk-transformations.test.ts:942
- Draft comment:
The new test suite for cache tokens in ai-sdk-transformations covers multiple edge cases, including overriding and string values. Excellent coverage. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
8. packages/traceloop-sdk/test/ai-sdk-transformations.test.ts:1964
- Draft comment:
Overall, the new attribute transformation tests (including telemetry metadata and tool calls) integrate well with existing functionality. No critical issues found. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
Workflow ID: wflow_NAbAr5c82Wm42HHN
You can customize
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed cc54052 in 1 minute and 37 seconds. Click for details.
- Reviewed
162lines of code in4files - Skipped
0files when reviewing. - Skipped posting
3draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:431
- Draft comment:
Good use of telemetry metadata to extract and attach agent info. Consider adding logging or error details in the catch blocks to help diagnose parsing issues in production. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
2. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:548
- Draft comment:
Span name transformation uses agent metadata for top-level spans. It might be helpful to clarify in comments that child spans retain their mapped names when agent metadata isn’t applied. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
3. packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts:397
- Draft comment:
When both 'ai.usage.cacheReadInputTokens' and 'ai.usage.cachedInputTokens' are present, the latter overwrites the former (last write wins). Confirm this behavior is intentional as per tests. - Reason this comment was not posted:
Confidence changes required:0%<= threshold50%None
Workflow ID: wflow_lEnRXLusaNUq0zNd
You can customize
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
packages/traceloop-sdk/test/ai-sdk-transformations.test.ts (1)
943-1170: Cache token tests look correct, but some names/comments are misleadingThe new cache token tests exercise all relevant paths (creation, read, OpenAI
cachedInputTokens, coexistence, zero/string values, and integration) and match the implementation of the cache transforms andcalculateTotalTokens.Two minor clarity issues you may want to tidy up:
- Test name vs behavior mismatch:
it("should prefer cacheReadInputTokens over cachedInputTokens when both present", ...)actually verifies thatcachedInputTokenswins (last write wins) and the inline comment also explains that. Renaming the test (e.g. to “should prefer cachedInputTokens when both present (last write wins)”) would avoid confusion.- Comment vs behavior mismatch:
- In
it("should not modify attributes when cache token attributes are not present", ...), the comment says “Should preserve input tokens and add total tokens”, but with only input tokens presentcalculateTotalTokensintentionally does not set a total. The assertions don’t depend on this, so either adjust or drop that part of the comment.These are cosmetic only; the assertions correctly reflect the current transform behavior.
packages/traceloop-sdk/test/ai-sdk-integration.test.ts (1)
273-446: Strong coverage for prompt-cache token transformation, with minor coupling to fixture token countThis test does a good job validating both directions of the cache token mapping: it forces a cacheable prompt, checks
gen_ai.usage.*input/output and cache-* attributes viaSpanAttributes, and asserts that the legacyai.usage.*cache keys are fully stripped from spans. Resetting the exporter between calls and guarding the cache-token assertions behind existence checks keeps it robust across providers. As per coding guidelines, usingSpanAttributesfor thegen_ai.*keys is exactly what we want here.One nit: the strict expectation of
6900cache-read tokens ties the test to a particular HAR/model/tokenizer. That’s fine for a recorded fixture, but if recordings are regenerated or models change you’ll need to update this constant. If you want to reduce that coupling, you could keep only the> 0check or assert a looser lower bound.packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (1)
7-24: Agent-aware span naming for top-level AI SDK spans looks correct; consider de-duplicating the top-level listThe new constants and
HANDLED_SPAN_NAMESentries forai.streamText,ai.generateObject, andai.streamObject, together withTOP_LEVEL_AI_SPANSand the updatedtransformAiSdkSpanNames, give you the intended behavior:
- Top-level AI spans (
ai.generateText/ai.streamText/ai.generateObject/ai.streamObject) are renamed to the agent name only whenai.telemetry.metadata.agentis present.- Child spans (
*.doGenerate/*.doStream) are always normalized totext.generate,object.generate,text.stream,object.stream, preserving the previous observability surface.This aligns with the PR’s goal and keeps non-agent spans from being renamed unexpectedly. As per coding guidelines, you’re correctly keeping semantic attribute strings out of here and delegating them to
SpanAttributeselsewhere.One small maintainability improvement:
transformTelemetryMetadatadefines its owntopLevelSpanNamesarray with the same four values asTOP_LEVEL_AI_SPANS. ReusingTOP_LEVEL_AI_SPANSthere would avoid the risk of future drift when new top-level operations are added.Also applies to: 540-565
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
Disabled knowledge base sources:
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (6)
packages/ai-semantic-conventions/src/SemanticAttributes.ts(1 hunks)packages/traceloop-sdk/recordings/Test-AI-SDK-Integration-with-Recording_156038438/should-capture-and-transform-cache-tokens-from-OpenAI-with-prompt-caching_4027203422/recording.har(1 hunks)packages/traceloop-sdk/recordings/Test-AI-SDK-Integration-with-Recording_156038438/should-capture-and-transform-cache-tokens-from-OpenAI-with-prompt-caching_4027203422/recording.har.backup(1 hunks)packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts(6 hunks)packages/traceloop-sdk/test/ai-sdk-integration.test.ts(1 hunks)packages/traceloop-sdk/test/ai-sdk-transformations.test.ts(2 hunks)
✅ Files skipped from review due to trivial changes (2)
- packages/traceloop-sdk/recordings/Test-AI-SDK-Integration-with-Recording_156038438/should-capture-and-transform-cache-tokens-from-OpenAI-with-prompt-caching_4027203422/recording.har
- packages/traceloop-sdk/recordings/Test-AI-SDK-Integration-with-Recording_156038438/should-capture-and-transform-cache-tokens-from-OpenAI-with-prompt-caching_4027203422/recording.har.backup
🧰 Additional context used
📓 Path-based instructions (3)
packages/ai-semantic-conventions/src/SemanticAttributes.ts
📄 CodeRabbit inference engine (CLAUDE.md)
Define all AI/LLM span attribute constants in packages/ai-semantic-conventions/src/SemanticAttributes.ts
Files:
packages/ai-semantic-conventions/src/SemanticAttributes.ts
packages/{instrumentation-*,traceloop-sdk}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
Import AI/LLM semantic attribute constants from @traceloop/ai-semantic-conventions rather than hardcoding strings
Files:
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.tspackages/traceloop-sdk/test/ai-sdk-transformations.test.tspackages/traceloop-sdk/test/ai-sdk-integration.test.ts
packages/traceloop-sdk/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
packages/traceloop-sdk/**/*.{ts,tsx}: Use the provided decorators (@workflow, @task, @agent) for workflow/task/agent spans instead of re-implementing them
For manual LLM operations, use trace.withLLMSpan from @traceloop/node-server-sdk
Files:
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.tspackages/traceloop-sdk/test/ai-sdk-transformations.test.tspackages/traceloop-sdk/test/ai-sdk-integration.test.ts
🧠 Learnings (6)
📓 Common learnings
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/ai-semantic-conventions/src/SemanticAttributes.ts : Define all AI/LLM span attribute constants in packages/ai-semantic-conventions/src/SemanticAttributes.ts
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/instrumentation-*/**/*.{ts,tsx} : Instrumentations must create spans with appropriate AI/LLM semantic attributes for calls they wrap
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/{instrumentation-*,traceloop-sdk}/**/*.{ts,tsx} : Import AI/LLM semantic attribute constants from traceloop/ai-semantic-conventions rather than hardcoding strings
📚 Learning: 2025-08-24T22:08:07.023Z
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/ai-semantic-conventions/src/SemanticAttributes.ts : Define all AI/LLM span attribute constants in packages/ai-semantic-conventions/src/SemanticAttributes.ts
Applied to files:
packages/ai-semantic-conventions/src/SemanticAttributes.tspackages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.tspackages/traceloop-sdk/test/ai-sdk-transformations.test.ts
📚 Learning: 2025-08-24T22:08:07.023Z
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/{instrumentation-*,traceloop-sdk}/**/*.{ts,tsx} : Import AI/LLM semantic attribute constants from traceloop/ai-semantic-conventions rather than hardcoding strings
Applied to files:
packages/ai-semantic-conventions/src/SemanticAttributes.tspackages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.tspackages/traceloop-sdk/test/ai-sdk-transformations.test.tspackages/traceloop-sdk/test/ai-sdk-integration.test.ts
📚 Learning: 2025-08-24T22:08:07.023Z
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/instrumentation-*/**/*.{ts,tsx} : Instrumentations must create spans with appropriate AI/LLM semantic attributes for calls they wrap
Applied to files:
packages/ai-semantic-conventions/src/SemanticAttributes.tspackages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.tspackages/traceloop-sdk/test/ai-sdk-transformations.test.ts
📚 Learning: 2025-08-24T22:08:07.023Z
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/traceloop-sdk/**/*.{ts,tsx} : For manual LLM operations, use trace.withLLMSpan from traceloop/node-server-sdk
Applied to files:
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.tspackages/traceloop-sdk/test/ai-sdk-transformations.test.ts
📚 Learning: 2025-08-24T22:08:07.023Z
Learnt from: CR
Repo: traceloop/openllmetry-js PR: 0
File: CLAUDE.md:0-0
Timestamp: 2025-08-24T22:08:07.023Z
Learning: Applies to packages/traceloop-sdk/**/*.{ts,tsx} : Use the provided decorators (workflow, task, agent) for workflow/task/agent spans instead of re-implementing them
Applied to files:
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.tspackages/traceloop-sdk/test/ai-sdk-transformations.test.ts
🧬 Code graph analysis (3)
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (1)
packages/ai-semantic-conventions/src/SemanticAttributes.ts (1)
SpanAttributes(23-78)
packages/traceloop-sdk/test/ai-sdk-transformations.test.ts (2)
packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (1)
transformLLMSpans(495-512)packages/ai-semantic-conventions/src/SemanticAttributes.ts (1)
SpanAttributes(23-78)
packages/traceloop-sdk/test/ai-sdk-integration.test.ts (1)
packages/ai-semantic-conventions/src/SemanticAttributes.ts (1)
SpanAttributes(23-78)
🔇 Additional comments (4)
packages/ai-semantic-conventions/src/SemanticAttributes.ts (1)
38-40: Cache token span attributes are consistent and correctly scopedThe new cache creation/read attributes follow the existing
gen_ai.usage.*convention and are correctly centralized inSpanAttributes, matching how downstream transforms consume them. This aligns with the guidance to define AI/LLM span attribute constants here.packages/traceloop-sdk/test/ai-sdk-transformations.test.ts (1)
2006-2008: Agent metadata test now accurately simulates production spanName flowPassing
"research_assistant"as thespanNameargument totransformLLMSpansmatches the real pipeline wheretransformAiSdkSpanNamesrenames the top-level span before attribute transformation. This keeps the agent root-span test aligned with runtime behavior.packages/traceloop-sdk/src/lib/tracing/ai-sdk-transformations.ts (2)
35-38: Cache token transformations correctly normalize AI SDK attributes to gen_ai cache usage attributesThe new cache-related constants and transformer functions cleanly map AI SDK–specific usage fields into the semantic
gen_ai.usage.*space:
ai.usage.cacheCreationInputTokens→SpanAttributes.LLM_USAGE_CACHE_CREATION_INPUT_TOKENS.ai.usage.cacheReadInputTokens→SpanAttributes.LLM_USAGE_CACHE_READ_INPUT_TOKENS.ai.usage.cachedInputTokens(OpenAI-specific) →SpanAttributes.LLM_USAGE_CACHE_READ_INPUT_TOKENS.Each transformer deletes the legacy
ai.usage.*key after mapping, andtransformLLMSpanscalls them beforecalculateTotalTokens, so cache tokens don’t interfere with the standard input/output token sum. This matches the semantics exercised in the new integration test and keeps all externally consumed keys on theSpanAttributessurface. Based on learnings, this is the right place to centralize these attribute mappings.Also applies to: 375-405, 506-508
466-483: Agent span-kind logic now correctly scopes AGENT to top-level AI spansThe updated
transformTelemetryMetadatabehavior—always settingSpanAttributes.GEN_AI_AGENT_NAMEwhen anagentmetadata key is present, but only markingTRACELOOP_SPAN_KIND = AGENT(andTRACELOOP_ENTITY_NAME) when the span is top-level (original AI span name or renamed to the agent) — cleanly distinguishes agent root spans from their children. This keeps downstream analysis from over-tagging nested LLM/tool spans while still propagating the agent name as context to all spans viaGEN_AI_AGENT_NAME. The logic is sound and lines up with the agent-aware naming introduced intransformAiSdkSpanNames.
| } | ||
| }; | ||
|
|
||
| const transformCacheCreationInputTokens = ( |
There was a problem hiding this comment.
@galkleinman maybe we can just transform any ai.usage. prefix to gen_ai.usage.?
There was a problem hiding this comment.
I think I prefer the explicity. it makes our life easier understanding what should be published and what not.
| if (span.name in HANDLED_SPAN_NAMES) { | ||
| span.updateName(HANDLED_SPAN_NAMES[span.name]); | ||
| // Check if this is a top-level AI span with agent metadata | ||
| const agentName = span.attributes[`${AI_TELEMETRY_METADATA_PREFIX}agent`]; |
There was a problem hiding this comment.
I think there's already a method for checking / extracting the agent name
| } | ||
| if (span.name in HANDLED_SPAN_NAMES) { | ||
| span.updateName(HANDLED_SPAN_NAMES[span.name]); | ||
| // Check if this is a top-level AI span with agent metadata |
There was a problem hiding this comment.
nit: redundant comment (this entire section is overly commented which reduces readability)
Important
Enhance AI SDK by integrating agent metadata into span naming and attributes, supporting
generateObjectandstreamTextoperations, and adding cache token transformations.generateObjectandstreamTextoperations with agent metadata.transformLLMSpans()inai-sdk-transformations.tsupdated to handle cache token transformations and agent metadata.transformCacheCreationInputTokens(),transformCacheReadInputTokens(), andtransformCachedInputTokens()added for cache token handling.ai-sdk-agent-integration.test.tsandai-sdk-integration.test.tsto verify agent name propagation and cache token transformations.recordings/to reflect new span naming and cache handling.This description was created by
for cc54052. You can customize this summary. It will automatically update as commits are pushed.
Summary by CodeRabbit
New Features
Tests
✏️ Tip: You can customize this high-level summary in your review settings.