ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstruction

Xinze Li, Yiyuan Wang, Pengxu Chen, Wentao Fan, Weifeng Su, Weisi Lin, Wentao Cheng

🔭 Overview

ReCal3R estimates state token reliability and a candidate learning rate from the recurrent forward pass, and uses reliability calibration to obtain the final state learning rate. This prevents unreliable state tokens from receiving aggressive updates and leads to cleaner reconstructions than CUT3R over long image streams.

Getting Started

Installation

1. Clone ReCal3R.

git clone https://github.com/Powertony102/ReCal3R.git
cd ReCal3R

2. Create the environment.

conda create -n recal3r python=3.11 cmake=3.14.0
conda activate recal3r
conda install pytorch torchvision pytorch-cuda=12.1 -c pytorch -c nvidia  # use the correct version of cuda for your system
pip install -r requirements.txt
# issues with pytorch dataloader, see https://github.com/pytorch/pytorch/issues/99625
conda install 'llvm-openmp<16'
# for evaluation
pip install evo
pip install open3d

3. Compile the cuda kernels for RoPE (as in CroCo v2).

cd src/croco/models/curope/
python setup.py build_ext --inplace
cd ../../../../

Download Checkpoints

CUT3R provide checkpoints trained on 4-64 views: cut3r_512_dpt_4_64.pth.

To download the weights, run the following commands:

cd src
gdown --fuzzy https://drive.google.com/file/d/1Asz-ZB3FfpzZYwunhQvNPZEUA8XUNAYD/view?usp=drive_link
cd ..

Evaluation

Please refer to the eval.md for more details.

🍺 Acknowledgements

• Thanks to these great repositories:CUT3R, TTT3R, TTSA3R, MeMix, Easi3R, DUSt3R, MonST3R, Spann3R, Viser, S-VGGT and many other inspiring works in the community.

• Special thanks to our supervisor Dr. Wentao Cheng for consistent suggestions and efforts to this work.