Multi-domain calibration

This is part of the Research Project 2022 of TU Delft (https://github.com/TU-Delft-CSE/Research-Project)

About the code

The source code of calibration_module: https://github.com/ethen8181/machine-learning

The main package we used to process the data and train the model: Keras, sklearn, pandas, numpy, scipy.

Python version: 3.8

References:

[1] Qiao, F., & Peng, X. (2021). Uncertainty-guided Model Generalization to Unseen Domains. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 6786–6796. https://doi.org/10.1109/CVPR46437.2021.00672

[2] Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., Lee, T., David, E., Stavness, I., Guo, W., Earnshaw, B. A., Haque, I. S., Beery, S., Leskovec, J., Kundaje, A., . . . Liang, P. (2021). WILDS: A Benchmark of in-the-Wild Distribution Shifts. arXiv:2012.07421 [cs].

[3] Torralba, A., & Efros, A. A. (2011). Unbiased look at dataset bias. CVPR 2011, 15211528. https://doi.org/10.1109/CVPR.2011.5995347

[4] Rosenfeld, E., Ravikumar, P., & Risteski, A. (2021). The Risks of Invariant Risk Minimization. arXiv:2010.05761 [cs, stat].

[5] Arjovsky, M., Bottou, L., Gulrajani, I., & Lopez-Paz, D. (2020). Invariant Risk Minimization. arXiv:1907.02893 [cs, stat].

[6] Kamath, P., Tangella, A., Sutherland, D. J., & Srebro, N. (n.d.). Does Invariant Risk Minimization Capture Invariance?, 10.

[7] Wald, Y., Feder, A., Greenfeld, D., & Shalit, U. (2022). On Calibration and Out-ofdomain Generalization. arXiv:2102.10395 [cs].

[8] Koyama, M., & Yamaguchi, S. (2021). When is invariance useful in an Out-of-Distribution Generalization problem ?

[9] Naeini, M. P., Cooper, G. F., & Hauskrecht, M. (n.d.). Obtaining Well Calibrated Probabilities Using Bayesian Binning, 7.

[10] Niculescu-Mizil, A., & Caruana, R. (2005). Predicting good probabilities with supervised learning. Proceedings of the 22nd International Conference on Machine Learning

• ICML ’05, 625–632. https://doi.org/10.1145/1102351.1102430

[11] Leathart, T., Frank, E., Holmes, G., & Pfahringer, B. (2018). Probability Calibration Trees.

[12] Zadrozny, B., & Elkan, C. (n.d.[a]). Transforming Classifier Scores into Accurate Multiclass Probability Estimates, 6.

[13] Zadrozny, B., & Elkan, C. (n.d.[b]). Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers, 9.

[14] Gulrajani, I., & Lopez-Paz, D. (2020). In Search of Lost Domain Generalization.

[15] Posocco, N., & Bonnefoy, A. (2021). Estimating Expected Calibration Errors. arXiv:2109.03480 [cs].

[16] Bröcker, J. (2008). Some Remarks on the Reliability of Categorical Probability Forecasts. Monthly Weather Review, 136(11), 4488–4502. https : / / doi . org / 10 . 1175 / 2008MWR2329.1

[17] Murphy, A. H., & Winkler, R. L. (1977). Reliability of Subjective Probability Forecasts of Precipitation and Temperature. Applied Statistics, 26(1), 41. https://doi.org/10. 2307/2346866

[18] Martínez-Camblor, P., & Corral, N. (2012). A general bootstrap algorithm for hypothesis testing. Journal of Statistical Planning and Inference, 142(2), 589–600. https: //doi.org/10.1016/j.jspi.2011.09.003