Support per-sample weights in progressive validation#1795
Merged
MaxHalford merged 6 commits intomainfrom Apr 9, 2026
Merged
Conversation
…sive_val_score Allow per-sample weights via two complementary APIs: - weights callable: progressive_val_score(..., weights=lambda x, y: float) - dataset triples: dataset yields (x, y, w) where w is a per-sample float Tuple weights take precedence over the callable so mixed datasets behave predictably. The weight is forwarded to learn_one for models that accept a w parameter (e.g. linear_model.LogisticRegression). Models without a w parameter are called without it to preserve backward compatibility. Implementation: - _needs_weights guard: weight infrastructure (weight_queue, _iter_dataset) is only created when the model accepts w or a weights callable is given, keeping the default path free of any overhead - weight_queue (collections.deque) bounds memory to the delay window, not the full dataset - kwargs w-key collision guard strips w from simulate_qa metadata before learn_one to prevent TypeError when stream metadata includes a w key Closes #1502
river's EstimatorMeta.__instancecheck__ accesses instance._last_step to unwrap pipelines. On a plain MagicMock every attribute access returns another MagicMock, so isinstance(mock, AnomalyFilter) recurses infinitely and hits Python's recursion limit. Setting _last_step=None on the test mock stops the chain: hasattr(None, '_last_step') is False so __instancecheck__ returns False immediately.
AnomalyFilter.learn_one(*args, **learn_kwargs) was being treated as accepting w because of the VAR_KEYWORD check, causing it to receive w=1.0 and silently forward it to OneClassSVM.learn_one which has no w parameter, raising TypeError. Fix _model_accepts_w to check only for an explicit w parameter in the signature, not **kwargs. Models that want per-sample weights should declare w explicitly. Also fix _make_model in the weight tests to use create_autospec instead of manually assigning __signature__ on a MagicMock, which causes RecursionError in Python 3.13. Update the no-w model test to use (x, y) pairs since (x, y, w_float) triples through the non-weight code path expect the third element to be a kwargs dict.
mypy infers preds as dict[int, tuple[Any, Any|bool, float]] from the 3-tuple assignment and then flags the 2-tuple assignment and 2-variable unpack as type errors. Annotate with typing.Any to allow both shapes.
Replace the weights callable, weight_queue, and duplicated fast path with a simpler approach: extract "w" from the existing kwargs dict in dataset tuples and forward it to learn_one. Also fix Pipeline.learn_one to forward **params to the final supervised step so weights reach the underlying model. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
progressive_val_score/iter_progressive_val_scorevia(x, y, {"w": 2.0})dataset tuplesPipeline.learn_oneto forward**paramsto the final supervised step so weights reach the underlying modelDetails
Weights are extracted from the existing kwargs dict mechanism (
"w"key is popped and forwarded tolearn_oneseparately). No new function parameters, no weight queue, no duplicated code paths. Models that don't acceptwgracefully ignore it.Test plan
wparam ignore weights gracefully🤖 Generated with Claude Code