feat(eval): Add span output attributes and metadata for evaluation spans

[AAgnihotry]

Summary

This PR adds comprehensive span output attributes and metadata for evaluation spans, following the AgentOutputSpanAttributes pattern with Pydantic models.

Changes

New Module: _span_utils.py

Created a centralized utility module for evaluation span configuration:

Pydantic Models:

• EvalSetRunOutput: Output model for "Evaluation Set Run" spans (score as int)
• EvaluationOutput: Output model for "Evaluation" spans (score as int)
• EvaluationOutputSpanOutput: Output model for "Evaluation output" spans (type, value, justification)

Calculation Functions:

• calculate_overall_score(): Calculates average across all evaluators
• calculate_evaluation_average_score(): Calculates average for a single evaluation

Low-level Attribute Setters:

• set_eval_set_run_output_and_metadata(): Sets output and metadata for eval set run spans
• set_evaluation_output_and_metadata(): Sets output and metadata for evaluation spans
• set_evaluation_output_span_output(): Sets output for evaluation output spans

High-level Configuration Functions:

• configure_eval_set_run_span(): Complete configuration including schema retrieval and score calculation
• configure_evaluation_span(): Complete configuration with error handling

Updated: _runtime.py

• Refactored to use utility functions from _span_utils.py
• Simplified code from ~30 lines to ~6 lines per span type
• All three span types now properly set output and metadata attributes

Span Attributes Added

All evaluation spans now include:

• output: JSON string containing score (for eval set run and evaluation) or type/value/justification (for evaluation output)
• agentId: execution ID
• agentName: "N/A"
• inputSchema: runtime input schema as JSON string
• outputSchema: runtime output schema as JSON string

Tests

Unit Tests (test_eval_span_utils.py): 19 new tests

• Pydantic model serialization tests
• Calculation function tests
• Low-level span attribute setting tests
• High-level configuration function tests with async/await

Integration Tests (test_eval_tracing_integration.py): 3 new tests

• Verification that "Evaluation Set Run" span has output with score
• Verification that "Evaluation" span has metadata attributes
• Verification that "Evaluation output" span has type, value, and justification

Span Attribute Tests (test_eval_runtime_spans.py): 13 new tests

• Tests for output attributes on all three span types
• Tests for metadata attributes (agentId, agentName, schemas)
• Tests for proper JSON structure and types

Test Infrastructure Fix:

• Fixed SpanCapturingTracer to capture attributes set via span.set_attribute() (not just initial attributes)

Test Results

• ✅ All 1531 tests passing (7 skipped for authentication)
• ✅ 19 new unit tests for span utilities
• ✅ 3 new integration tests for span attributes
• ✅ 13 new span attribute tests
• ✅ Linting: ruff check passed
• ✅ Formatting: ruff format passed
• ✅ Type checking: mypy passed

Files Changed

• src/uipath/_cli/_evals/_span_utils.py (new, 290 lines)
• src/uipath/_cli/_evals/_runtime.py (40 lines modified)
• tests/cli/eval/test_eval_span_utils.py (new, 462 lines)
• tests/cli/eval/test_eval_runtime_spans.py (184 lines added)
• tests/cli/eval/test_eval_tracing_integration.py (297 lines added)

Total: 1,268 lines added, 5 lines removed

🤖 Generated with Claude Code

Development Package

• Use uipath pack --nolock to get the latest dev build from this PR (requires version range).
• Add this package as a dependency in your pyproject.toml:

[project]
dependencies = [
  # Exact version:
  "uipath==2.4.9.dev1010793680",

  # Any version from PR
  "uipath>=2.4.9.dev1010790000,<2.4.9.dev1010800000"
]

[[tool.uv.index]]
name = "testpypi"
url = "https://test.pypi.org/simple/"
publish-url = "https://test.pypi.org/legacy/"
explicit = true

[tool.uv.sources]
uipath = { index = "testpypi" }

[tool.uv]
override-dependencies = [
    "uipath>=2.4.9.dev1010790000,<2.4.9.dev1010800000",
]