Skip to content

Statistical model comparison with bootstrap and beyond

License

Notifications You must be signed in to change notification settings

Oulu-IMEDS/stambo

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

38 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

StaMBO: Statistical model comparison with bootstrap

DOI PyPI version docs License

This package is aimed to be a one-stop-shop for statistical testing in machine learning when it comes to evaluating models on a test set and comparing whether our improved model is really beating the baseline. That is, we cover the following very typical use-case in machine learning: usecase

Currently, we support the cases of classification, regresson, and semantic segmentation. We do not yet support the significance of ranking, as well as grouped data. It is coming in the future releases.

In practice

Install from PyPI:

pip install stambo

The use of the library is then straightforward:

import stambo
...
seed = 42
testing_result = stambo.compare_models(y_test, preds_1, preds_2, metrics=("ROCAUC", "AP", "QKappa", "BACC", "MCC"), seed=seed)
print(stambo.to_latex(testing_result))

The above will print a LaTeX table, which one can easily copy-paste. As an example, below is the rendered table, which was returned in notebooks/Classification.ipynb (Binder): Table

The regression example can be found at notebooks/Regression.ipynb (Binder )

For more advanced explanation, see the documentation. By default, binary, multi-class, and multi-label classification, as well as regression are supported.

One can also use the library to perform a simple two-sample test. For example, to compare the means of two distributions:

import stambo
...
seed = 42
res = stambo.two_sample_test(sample_1, sample_2, statistics={"Mean": lambda x: x.mean()})

A more detailed and full example of the above is shown here: notebooks/Two_sample_test.ipynb (Binder)

Contributing

To setup a dev environment, you should use the provided environment file, and compile the documentation locally:

conda env create -f env.yaml
conda activate stambo-dev
pip install -e .
cd docs
make html

Author

Dr. Aleksei Tiulpin, PhD