Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Multiple Model Comparison Test.

This repository contains resources for finding the best model with multiple model comparison. The algorithm is described in our paper,

Kernel Stein Tests for Multiple Model Comparison
Jen Ning Lim, Makoto Yamada, Bernhard Schölkopf, Wittawat Jitkrittum
NeurIPS 2019

How to install?

Requires numpy, autograd, matplotlib and SciPy. The package can be installed with pip command.

pip install -e /path/to/the/folder/of/this/repo/after/clone

Or alternatively,

pip install git+https://github.com/jenninglim/model-comparison-test.git

Once installed, you should be able to do import reltest without any error.

Demo

See notebooks/demo_reltest.ipynb.

Reproducing results

See reproduce-results.

Disclaimer

The current implementation relies on an accurate approximation of the inverse CDF in the tail regions of the truncated normal (see 6). The implementation uses the inverse CDF to calculate the rejection threshold. If the test statistic is greater than the rejection threshold, we reject the null hypothesis.

The p-values produced from the modules may not be accurate. At the time of writting this code there were several problems with the scipy.stats.truncnorm module. See 1, 2, 3 and 4. This may have been fixed in the pull request 5 in SciPy 1.4.0.

See also

About

NeurIPS 2019. Kernel Stein Tests for Multiple Model Comparison.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published