For all synthetic exp: compare coverage and efficiency vs number of samples
Metric: error rate withing subgroups
- (TODO: implement error rate metric per subgroup)
- (TODO: APIs for all baselines - Run experiment(baseline, data) API)
- (TODO: Run experiments for all baselines in each setting)
- Standard CP [Done]
- Quantile regression [Done~]
- Local CP (CP within reported subgroups) [Done]
- Conformalized quantile regression: https://github.com/yromano/cqr [Done~]
- Conditional histograms: https://github.com/msesia/chr [Done~]
- LCP (https://arxiv.org/abs/2106.08460)
- Baselines from: https://github.com/msesia/cqr-comparison
- Locally adaptive conformal prediction: https://github.com/yromano/cqr [Done~]
- Orthogonal quantile regression https://github.com/Shai128/oqr
Metric: efficiency within subgroups
TCP-RIF vs TCP-baselines
Data sets: Synthetic / UCI / Breast cancer