This repository is the official implementation of
Causal Triplet: An Open Challenge for Intervention-centric Causal Representation Learning
Conference on Causal Learning and Reasoning (CLeaR), 2023
Yuejiang Liu, Alexandre Alahi, Chris Russell, Max Horn, Dominik Zietlow, Bernhard Schölkopf, Francesco Locatello

If you have any questions regarding the code/dataset, please feel free to raise issues in the repo or email me at yuejiang.liu[at]epfl.ch.

Requirements

To install requirements:

pip install -r requirements.txt

Dataset

To generate paired images from ProcTHOR:

python procthor/generator.py --min_scene_idx=0 --max_scene_idx=9999

Below are some examples of the generated image pairs, where an 'open' action is performed:

           

The total size of the dataset is ~165GB. Examples collected from ProcTHOR room #0-2000 can be downloaded at zenodo.

Training

1. Single-object images under compositional distribution shifts

   To run the experiments encouraging independence between action class and object class:

   bash script/run_comp_critic.sh

   To run the experiments encouraging block-disentangled visual representations with the sparsity regularizer:

   bash script/run_comp_sparse.sh

2. Single-object images under systematic distribution shifts

   To run the experiments encouraging independence between action class and object class:

   bash script/run_syst_critic.sh

   To run the experiments encouraging block-disentangled visual representations with the sparsity regularizer:

   bash script/run_syst_sparse.sh

3. Simulated multi-object images under systematic distribution shifts

   To run the experiments approximating object-centric representations with instance masks:

   bash script/run_instance_mask.sh

   To run the experiments exploiting the latent structures in the Slot Attention with different matching modules:

   bash script/run_slot_thor.sh

   To run the foundation model for segmentation (SAM) with a point prompt:

   bash script/run_sam_thor.sh

4. Real-world multi-object images under compositional distribution shifts

   To run the experiments exploiting the latent structures in GroupVIT with different matching modules:

   bash script/run_group_epic.sh

Analysis

To analyze experiment results:

bash script/run_analysis.sh

Examples of experiment results from the saved logs:

• Effect of block-disentanglement on single-object images under compositional distribution shifts (left: ID, right: OOD)

• Effect of approximate object-centric representations on simulated multi-object images (left: ID, right: OOD)

• Effect of exploiting group structures on real-world multi-object images (left: ID, right: OOD)

• Effect of exploiting slot structures on simulated multi-object images (left: ID, right: OOD)

• Visualization of implicit Slot Attention on simulated multi-object images

• Visualization of segmentation anything model (SAM) on simulated multi-object images. Three masks of the highest scores are visualized, given a point prompt in the shadow.

Citation

If you find this code useful for your research, please cite our paper:

@inproceedings{Liu2023CausalTriplet,
  title={Causal Triplet: An Open Challenge for Intervention-centric Causal Representation Learning},
  author={Liu, Yuejiang and Alahi, Alexandre and Russell, Chris and Horn, Max and Zietlow, Dominik and Sch{\"o}lkopf, Bernhard and Locatello, Francesco},
  booktitle={2nd Conference on Causal Learning and Reasoning (CLeaR)},
  year={2023}
}