Description
The SDK’s OneOrMoreScores type currently includes list[ScoreLike] as the multi-score return type. Because Python lists are invariant / mutable, static type checkers reject scorers that return list[Score].
Reproduction
Run pyright on the following code:
from braintrust import Score
from braintrust.framework import EvalScorer
def scorer(input: str, output: str, expected: str | None = None) -> list[Score]:
return [Score(name="match", score=1.0)]
typed_scorer: EvalScorer[str, str, str] = scorer
Expected
It should allow you to return a list[Score].
Observed
You get a type error on line 7:
Function return type "list[Score]" is incompatible with type "OneOrMoreScores"
Description
The SDK’s
OneOrMoreScorestype currently includeslist[ScoreLike]as the multi-score return type. Because Python lists are invariant / mutable, static type checkers reject scorers that returnlist[Score].Reproduction
Run pyright on the following code:
Expected
It should allow you to return a
list[Score].Observed
You get a type error on line 7:
Function return type "list[Score]" is incompatible with type "OneOrMoreScores"