McMetric

This repository provides software that can automatically reproduce all results in the associated paper, "Measuring multi-calibration," authored by Ido Guy, Daniel Haimovich, Fridolin Linder, Nastaran Okati, Lorenzo Perini, Niek Tax, and Mark Tygert.

Unit tests

To test the Kuiper metrics and tree traversals, issue the following commands on the command-line from the root directory of McMetric:

cd tests; PYTHONPATH=${PYTHONPATH:+${PYTHONPATH}:}.. pytest test_kuiper.py; \
    cd ..
cd tests; PYTHONPATH=${PYTHONPATH:+${PYTHONPATH}:}.. pytest test_traverse.py; \
    cd ..

Reproducing the paper

To process the data sets and generate corresponding LaTeX source for tables, issue the following commands on the command-line from the root directory:

cd analysis; gunzip cup98lrn.txt.gz; cd ..
cd analysis; gunzip psam_h06.csv.gz; cd ..
cd analysis; PYTHONPATH=${PYTHONPATH:+${PYTHONPATH}:}.. python acs.py \
    --response_var TABLET --big; cd ..
cd analysis; python big.py > big.out; cd ..
cd analysis; PYTHONPATH=${PYTHONPATH:+${PYTHONPATH}:}.. python acs.py \
    --response_var TABLET; cd ..
cd analysis; PYTHONPATH=${PYTHONPATH:+${PYTHONPATH}:}.. python acs.py \
    --response_var LAPTOP; cd ..
cd analysis; PYTHONPATH=${PYTHONPATH:+${PYTHONPATH}:}.. python acs.py \
    --response_var SMARTPHONE; cd ..
cd analysis; PYTHONPATH=${PYTHONPATH:+${PYTHONPATH}:}.. python kddcup.py; \
    cd ..
cd analysis; python tables.py; cd ..
cd analysis; PYTHONPATH=${PYTHONPATH:+${PYTHONPATH}:}.. python acssub.py \
    > acssub.out; cd ..

Credits

Essentially everything not related to the "multi" in "multi-calibration" is due to Matt Muckley and Narine Kokhlikyan. The remainder is due to the co-authors of the associated paper, "Measuring multi-calibration."

Copyright license

This McMetric software is licensed under the LICENSE (the MIT License) file in the root directory of this source tree.