Skip to content

fix: use Mapping instead of dict for EvalResult.scores#360

Merged
Abhijeet Prasad (AbhiPrasad) merged 1 commit intomainfrom
fix/eval-result-scores-mapping
Apr 28, 2026
Merged

fix: use Mapping instead of dict for EvalResult.scores#360
Abhijeet Prasad (AbhiPrasad) merged 1 commit intomainfrom
fix/eval-result-scores-mapping

Conversation

@starfolkai
Copy link
Copy Markdown
Contributor

@starfolkai starfolkai Bot commented Apr 28, 2026

Summary

EvalResult.scores is typed as dict[str, float | None], but dict is invariant in its value type, so callers constructing an EvalResult with a homogeneous dict[str, float] (no None values) get a type error from both pyright and mypy:

error: Argument of type "dict[str, float]" cannot be assigned to parameter "scores"
  of type "dict[str, float | None]" in function "__init__"
  Type parameter "_VT@dict" is invariant, but "float" is not the same as "float | None"
  Consider switching from "dict" to "Mapping" which is covariant in the value type

This PR widens the annotation from dict[str, float | None] to Mapping[str, float | None]. The framework only iterates result.scores.items() in build_local_summary and never mutates scores after construction, so Mapping is the correct abstraction. Same fix pattern as #281 (EvalParameters) and #285 (Metadata / ParametersSchema). Mapping is already imported.

Fixes #292.

Test plan

  • pyright on the issue's repro snippet: 0 errors, 0 warnings
  • mypy on the issue's repro snippet: Success: no issues found
  • pyright + mypy on src/braintrust/type_tests/: clean
  • pytest src/braintrust/test_framework.py: 35 passed, 1 skipped
  • ruff check + pre-commit hooks on framework.py: clean
  • End-to-end Eval(...) against the live Braintrust API with mixed float / None scorer outputs: results and summary unchanged ({'exact_match': 1.0, 'always_none': None}, summary 100%)

🤖 Generated with Claude Code

`EvalResult.scores` was typed as `dict[str, float | None]`, but `dict`
is invariant in its value type. Callers constructing the dataclass with
a homogeneous `dict[str, float]` (no None values) get a type error from
both pyright and mypy.

The framework only iterates `result.scores.items()` (in
`build_local_summary`) and never mutates `scores` after construction,
so widening the annotation to `Mapping[str, float | None]` is safe and
matches the pattern already applied to `EvalParameters` (#281) and
`Metadata`/`ParametersSchema` (#285).

Fixes #292

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@AbhiPrasad Abhijeet Prasad (AbhiPrasad) merged commit 30fd9eb into main Apr 28, 2026
82 checks passed
@AbhiPrasad Abhijeet Prasad (AbhiPrasad) deleted the fix/eval-result-scores-mapping branch April 28, 2026 14:15
Abhijeet Prasad (AbhiPrasad) added a commit that referenced this pull request Apr 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

EvalResult.scores: dict invariance rejects dict[str, float] in constructor

1 participant