RCTrans: Transparent Object Reconstruction in Natural Scene via Refractive Correspondence Estimation

This is the official implementation of the paper "RCTrans: Transparent Object Reconstruction in Natural Scene via Refractive Correspondence Estimation". [SIGGRAPH Asia 2025]

Project page | Paper

TODO

We will release all the codes to facilitate the community. For now, we have organized the code for the object reconstruction. But the codes for RCNet training/testing and codes for creating the training datasets involve complex frameworks. We are still organizing them to facilitate the reproduction with the least amount of configuration. We would try to release them all in Jan. 2026.

• release reconstruction code
• release network training and testing code
• release calling RCNet interface in reconstruction code
• release dataset creation code
• release real data process code and guidance
• release reconstruction data
• release RCNet training and validation dataset

Setup

1. Install torch and torchvision according to your environment. For reference, we use torch==2.0.1+cu118.
2. Install other packages.

   git clone https://github.com/arkgao/RCTrans.git
   cd RCTrans
   pip install -r requirements.txt

Dataset and Dataset Creation

We now release both the rendered dataset and the dataset creation code used for the synthetic correspondence data in this project.

• Rendered dataset: https://huggingface.co/datasets/arkgao/NaturalRefractiveCorrespondence
• Dataset creation repository: https://github.com/arkgao/RCDatasetCreation

If you only want to train or evaluate RCNet, download the dataset from Hugging Face and follow the instructions in ./RCEstimate/README.md.

If you want to reproduce the synthetic data generation pipeline, use the dedicated dataset creation repository. It contains the rendering code, resource preparation scripts, and guidance for preparing shape and environment-map assets before launching the batch renderer.

Object Reconstruction

All code related to object reconstruction is placed in the TransRecon directory. It would leverage the pretrained RCNet to reconstruct transparent objects from multi-view images.

Please first switch to this directory

cd TransRecon

and then follow the instructions provided in ./TransRecon/README.md to run the code.

RCNet Training & Testing

All code related to RCNet is placed in the RCEstimate directory. It includes the dataset description and all code.

Please first switch to this directory

cd RCEstimate

and then follow the instructions provided in ./RCEstimate/README.md to run the code.

Real Data Processing

We provide the guidance of processing real data in the RealDataProcess directory. It includes our code and the illustration of processing ./RealDataProcess/README.md.

Citation

@inproceedings{10.1145/3757377.3763859,
author = {Gao, Fangzhou and Kang, Yuzhen and Zhang, Lianghao and Wang, Li and Wang, Qishen and Zhang, Jiawan},
title = {RCTrans: Transparent Object Reconstruction in Natural Scene via Refractive Correspondence Estimation},
year = {2025},
url = {https://doi.org/10.1145/3757377.3763859},
doi = {10.1145/3757377.3763859},
booktitle = {Proceedings of the SIGGRAPH Asia 2025 Conference Papers},
articleno = {1},
numpages = {11},
}