Skip to content
/ chr Public

Conformal Histogram Regression: efficient conformity scores for non-parametric regression problems

Notifications You must be signed in to change notification settings

msesia/chr

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 
 
 

Repository files navigation

Conformal histogram regression

This repository contains a Python implementation of the conformal prediction method described in the accompanying paper: https://arxiv.org/abs/2105.08747

Matteo Sesia, Yaniv Romano
"Conformal Prediction using Conditional Histograms"
NeurIPS 2021 (spotlight)

Overview

Conformal histogram regression (CHR) computes prediction intervals for non-parametric regression that can automatically adapt to skewed data. It leverages black-box machine learning algorithms to estimate the conditional distribution of the outcome using histograms, and then translates their output into the shortest prediction intervals with approximate conditional coverage. The theoretical results presented in the paper prove the resulting prediction intervals have marginal coverage in finite samples, while asymptotically achiev- ing conditional coverage and optimal length if the black-box model is consistent. The code included here was utilized to carry out the numerical experiments with simulated and real data described in the paper, which demonstrate empirically the improved performance of CHR compared to state-of-the-art alternatives, including conformalized quantile regression and other distributional conformal prediction approaches.

Dependencies

This code is written for Python (v 3.7.6) and makes use of the following packages:

  • torch (v 1.6.0)
  • numpy (v 1.18.5)
  • pandas (v 1.1.0)
  • scipy (v 1.4.1)
  • six (v 1.15.0)
  • sklearn (v 0.23.2)
  • skgarden (v 0.1.2)
  • rpy2 (v 3.3.5) (optional and requires R installation with BART package, not used in the paper)

The tutorial notebook in the "examples/" directory is written for Jupyter and was originally compiled with the following setup.

  • jupyter core : 4.6.3
  • jupyter-notebook : 6.0.3
  • qtconsole : not installed
  • ipython : 7.13.0
  • ipykernel : 5.2.0
  • jupyter client : 6.1.2
  • jupyter lab : not installed
  • nbconvert : 5.6.1
  • ipywidgets : 7.6.3
  • nbformat : 5.0.4
  • traitlets : 4.3.3

This repository also included code to process the experimental results and produce the figures shown in the paper. This is written in R (v 4.0.3) and relies on the tidyverse package (v 1.3.0).

Instructions

Our method is implemented in the package contained within the "chr/" directory. This can be loaded and utilized as demonstrated in the tutorial notebook "examples/intro.ypynb".

The Python code needed to reproduce our numerical experiments are in the "experiments/" directory, along with bash scripts to submit the experiments, either sequentially (default), or on a computing cluster with a slurm interface. The script "experiments/dataset.py" loads and pre-processes the real data sets, which can be dowloaded freely from the sources referenced in the accompanying paper.

About

Conformal Histogram Regression: efficient conformity scores for non-parametric regression problems

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published