Problem
The parameters argument on Eval(), EvalAsync(), and the Evaluator dataclass is typed as:
parameters: dict[str, PromptParameter | ModelParameter | type[object] | None] | RemoteEvalParameters | None = None
dict is invariant in both its key and value types. This means callers who construct a dict containing only ModelParameter values get a mypy error:
from braintrust import Eval
from braintrust.parameters import ModelParameter
params: dict[str, ModelParameter] = {
"model": {"type": "model", "default": "openai:gpt-4o"},
}
Eval(
"my-eval",
data=...,
task=...,
scores=[...],
parameters=params, # error: Argument "parameters" has incompatible type
# "dict[str, ModelParameter]"; expected
# "dict[str, PromptParameter | ModelParameter | ...]"
)mypy even suggests the fix in its error note:
"dict" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
Consider using "Mapping" instead, which is covariant in the value type
Proposed fix
Change the annotation from dict to Mapping (from collections.abc). Mapping is covariant in the value type, so Mapping[str, ModelParameter] is assignable to Mapping[str, ModelParameter | PromptParameter | ...].
I confirmed that the SDK only ever reads the parameter dict (via .items(), .values(), .get()). It is never mutated after being passed in — _validate_local_parameters, is_eval_parameter_schema, and serialize_eval_parameters all build new result dicts rather than writing to the input. So Mapping is both correct and strictly more permissive.
The affected locations are:
Eval() function signature (framework.py)EvalAsync() function signature (framework.py)Evaluator dataclass field (framework.py)EvalParameters type alias (parameters.py)
Happy to open a PR if you'd prefer.
Problem
The
parametersargument onEval(),EvalAsync(), and theEvaluatordataclass is typed as:dictis invariant in both its key and value types. This means callers who construct a dict containing onlyModelParametervalues get a mypy error:mypy even suggests the fix in its error note:
Proposed fix
Change the annotation from
dicttoMapping(fromcollections.abc).Mappingis covariant in the value type, soMapping[str, ModelParameter]is assignable toMapping[str, ModelParameter | PromptParameter | ...].I confirmed that the SDK only ever reads the parameter dict (via
.items(),.values(),.get()). It is never mutated after being passed in —_validate_local_parameters,is_eval_parameter_schema, andserialize_eval_parametersall build new result dicts rather than writing to the input. SoMappingis both correct and strictly more permissive.The affected locations are:
Eval()function signature (framework.py)EvalAsync()function signature (framework.py)Evaluatordataclass field (framework.py)EvalParameterstype alias (parameters.py)Happy to open a PR if you'd prefer.