feat(vercel-ai): Add rerank support and fix token attribute mapping#19144
feat(vercel-ai): Add rerank support and fix token attribute mapping#19144
Conversation
- Add support for ai.rerank and ai.rerank.doRerank spans - Map parent span token attributes (ai.usage.inputTokens/outputTokens) to gen_ai.usage.input_tokens/output_tokens - Map ai.response.avgOutputTokensPerSecond to expected attribute name Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Enable automatic telemetry for the `rerank` method in the Node.js Vercel AI integration. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Codecov Results 📊✅ 535 passed | ⏭️ 51 skipped | Total: 586 | Pass Rate: 91.3% | Execution Time: 14m 35s All tests are passing successfully. Generated by Codecov Action |
size-limit report 📦
|
node-overhead report 🧳Note: This is a synthetic benchmark with a minimal express app and does not necessarily reflect the real-world performance impact in an application.
|
Methods like `rerank` were added in AI SDK v6, but the instrumentation supports versions >=3.0.0 <7. Without this check, `new Proxy(undefined, ...)` would throw a TypeError and crash all instrumentation on older SDK versions. Also adds E2E test app for AI SDK v6 to test rerank functionality. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
MockLanguageModelV3 requires content array instead of text field. MockRerankingModelV3 uses results/documentIndex/relevanceScore format. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
| ); | ||
| expect(rerankPipelineSpan).toBeDefined(); | ||
| expect(rerankPipelineSpan?.data?.['gen_ai.request.rerank.documents_count']).toBe(3); | ||
| expect(rerankPipelineSpan?.data?.['gen_ai.response.rerank.top_score']).toBe(0.95); |
There was a problem hiding this comment.
E2E test expects unmapped rerank attributes
Medium Severity
The E2E test expects rerank-specific attributes under the gen_ai.* namespace (gen_ai.request.rerank.query, gen_ai.request.rerank.documents_count, gen_ai.response.rerank.top_score), but the production code doesn't include any attribute mappings for rerank-specific data. Based on the existing codebase pattern, unmapped ai.* attributes get prefixed with vercel. and become vercel.ai.*. This mismatch means either the test expectations are incorrect (should use vercel.ai.*) or the production code is missing attribute mappings for rerank. Per the review rules, this flags that the test may not properly test the newly added behavior.
The E2E test for AI SDK v6 rerank functionality cannot work because of a bug in the AI SDK's MockRerankingModelV3. When calling rerank() with MockRerankingModelV3, the SDK crashes with: `TypeError: Cannot read properties of undefined (reading 'total')` This happens because the AI SDK's internal telemetry code expects usage data that the mock model doesn't provide (and isn't required by the RerankingModelV3 interface). The rerank instrumentation code is still fully functional and covered by unit tests in packages/core/test/lib/tracing/vercel-ai-rerank.test.ts. Note: Rerank mocking isn't available for E2E tests until the AI SDK fixes MockRerankingModelV3. Track: https://github.com/vercel/ai/issues Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
RulaKhaled
left a comment
RulaKhaled
left a comment
There was a problem hiding this comment.
Overall looks good, let's jus test this API in dev-packages/node-integration-tests/suites/tracing/vercelai/v6
RulaKhaled
left a comment
RulaKhaled
left a comment
There was a problem hiding this comment.
We chatted async, it seems the mock client provided by Vercel AI hasn’t implemented doRerank yet: https://github.com/vercel/ai/blob/main/packages/ai/src/test/mock-reranking-model-v3.ts#L15
lgtm, thank you
2 participants
Summary
This PR adds support for the Vercel AI SDK
rerankoperation and fixes token attribute mapping to follow OpenTelemetry semantic conventions.Changes
Core (
@sentry/core):ai.rerankspan support in the Vercel AI span processorai.usage.inputTokens/outputTokens→gen_ai.usage.input_tokens/output_tokensai.response.avgOutputTokensPerSecond→ standardized attribute nameNode (
@sentry/node):reranktoINSTRUMENTED_METHODSto auto-enable telemetry for rerank operationsWhy No E2E Tests for Rerank
E2E tests for rerank functionality cannot be added due to a bug in the AI SDK's
MockRerankingModelV3. When callingrerank()with the mock model, the SDK crashes with:This happens because the AI SDK's internal telemetry code expects usage data that the mock model doesn't provide (and isn't required by the
RerankingModelV3interface). The rerank instrumentation is fully covered by unit tests instead.See: https://github.com/vercel/ai/blob/main/packages/ai/src/test/mock-reranking-model-v3.ts
Checklist
yarn lint) & (yarn test).Closes #19145 (added automatically)