-
Notifications
You must be signed in to change notification settings - Fork 0
FAQ
Streams are evaluated prequentially: every instance is predicted before it is learned from, so every prediction is already on unseen data. The cumulative metric over the whole stream is the score, and the windowed series is the learning curve. Splitting a stream would destroy the temporal order that the drift lives in.
predict_one may return None while a model is warming up. Those instances
are counted but not scored. A handful at the start of a stream is normal; a
large number means the model never became ready — check the warm-up logic.
Either the method is deterministic, or reset() is not clearing the random
state. BenchmarkSuite sets model.seed before every run, but a framework
that builds its RNG once in __init__ and never rebuilds it in reset() will
ignore that.
Almost always leftover state: reset() did not fully return the model to its
untrained form. The test for it is in Extending SAMLB — run the same
stream twice with a reset() between and compare.
The compiled extension is stale. An editable install does not recompile when C++ changes; rebuild:
cmake -S . -B build/local -DCMAKE_BUILD_TYPE=Release
cmake --build build/local -j
cp build/local/_samlb_core.*.so samlb/No. Any iterable of (x_dict, y) works with any model — see Datasets.
BenchmarkSuite's tables and JSON output, though, are built on the dataset
registry, so a stream you want in the suite has to be added as an .npz.
For regression, yes — gradient-based learners diverge on raw target ranges. It min-max scales features using only statistics seen so far, so it leaks no future information. Regression targets are separately z-score normalised by the evaluator by default, with predictions mapped back before scoring, so reported errors are always in the original units.
Hours, for all frameworks × 15 datasets × 10 runs. Develop against
max_samples=10_000 and a couple of datasets, then run the full sweep once.
examples/run_benchmark.py --parallel --cpu_utilization 0.8 spreads seeds
across cores.
No. Exceptions are caught per run; the traceback lands on RunResult.error,
the table shows ERROR for that row, and aggregates exclude failed runs
instead of counting them as zeros.
Everything SAMLB used from River — scalers, feature selection, metrics, ADWIN/EDDM, ARF — is now native C++, verified bit-identical to River's implementations before the dependency was dropped. River is still fully usable as an optional backend through the adapters; see External Algorithms.
Not directly — scikit-learn estimators are batch learners with no learn_one.
Wrap one in a class that implements the three-method contract (buffering and
refitting periodically, say), or use River's or CapyMOA's streaming
equivalents through the adapters.
Accuracy for classification, R² for regression
(RunResult.primary_metric()). All of accuracy / f1 / precision /
recall and mae / rmse / r2 are recorded regardless; F1, precision and
recall are macro-averaged.
CapyMOA starts a JVM at import time. Check java -version — a JDK 17+ must be
on the path before pip install capymoa. CapyMOAClassifier.is_available()
returns False rather than raising when the JVM cannot start, so a suite can
skip it.
benchmark_results/ holds the recorded sweeps (CSV plus JSON summaries), and
results/ holds per-run JSON. See Benchmark API for the layout.