Document trace- and session-level eval scope on the eval submission API#36873
Merged
Conversation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
Preview links (active after the
|
brett0000FF
approved these changes
May 20, 2026
Contributor
brett0000FF
left a comment
brett0000FF
left a comment
There was a problem hiding this comment.
Approved but please see my comment about the paragraph that refers to the deleted tag.
| - **`trace`**: Associates the evaluation with an entire trace, identified by `join_on` pointing to the root span. | ||
| - **`session`**: Associates the evaluation with a session, identified by `session_id`. Do not include `join_on` for session-scoped evaluations. | ||
|
|
||
| To submit evaluations for <a href="/llm_observability/instrumentation/otel_instrumentation">OpenTelemetry spans</a> directly to the Evaluations API, you must include the <code>source:otel</code> tag in the evaluation. Additionally, <code>span_id</code> and <code>trace_id</code> values must be provided as **decimal** strings. If your OpenTelemetry instrumentation produces hexadecimal IDs, convert them to decimal before submitting. For example, in Python: <code>str(int(hex_span_id, 16))</code>. |
Contributor
There was a problem hiding this comment.
Should this be removed since we removed source:otel from the example?
Contributor
Author
There was a problem hiding this comment.
I reverted the Example section back to how it was originally. This was the first pass by claude that I pushed up here, so that I could get the preview links, but reviewing it now I think it's fine to just add a mention here about the new eval_scope field, with more details on the API page.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…s to API page Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
FouadWahabi
approved these changes
May 21, 2026
brett0000FF
approved these changes
May 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do? What is the motivation?
The eval submission API (
/api/intake/llm-obs/v2/eval-metric) now supports trace-level and session-level evaluations via a neweval_scopefield. This PR documents the new field and its behavior.Changes:
eval_scopeandsession_idto theEvalMetricreference table in the HTTP API referencejoin_onto reflect that it is now required only forspanandtracescope (notsession)Merge instructions
Merge readiness:
Additional notes
Backend changes: DataDog/dd-source#409393