Global-Aware Monocular Semantic Scene Completion with State Space Models (ICCV2025)

Abstract

Monocular Semantic Scene Completion (MonoSSC) reconstructs and interprets 3D environments from a single image, enabling diverse real-world applications. However, existing methods are often constrained by the local receptive field of Convolutional Neural Networks (CNNs), making it challenging to handle the non-uniform distribution of projected points (Fig. 1) and effectively reconstruct missing information caused by the 3D-to-2D projection. In this work, we introduce GA-MonoSSC, a hybrid architecture for MonoSSC that effectively captures global context in both the 2D image domain and 3D space. Specifically, we propose a DualHead Multi-Modality Encoder, which leverages a Transformer architecture to capture spatial relationships across all features in the 2D image domain, enabling more comprehensive 2D feature extraction. Additionally, we introduce the Frustum Mamba Decoder, built on the State Space Model (SSM), to efficiently capture long-range dependencies in 3D space. Furthermore, we propose a frustum reordering strategy within the Frustum Mamba Decoder to mitigate feature discontinuities in the reordered voxel sequence, ensuring better alignment with the scan mechanism of the State Space Model (SSM) for improved 3D representation learning. We conduct extensive experiments on the widely used Occ-ScanNet and NYUv2 datasets, demonstrating that our proposed method achieves state-of-the-art performance, validating its effectiveness. The code will be released upon acceptance.

Preparing GA-MonoSSC

Installation

1. Create conda environment:

$ conda create -n ga_monossc python=3.9 -y
$ conda activate ssc

2. This code was implemented with python 3.9, pytorch 2.0.0 and CUDA 11.7. Please install PyTorch:

$ conda install pytorch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 pytorch-cuda=11.8 -c pytorch -c nvidia

3. Install the additional dependencies:

$ git clone --recursive git@github.com:sj-li/GA-MonoSSC.git
$ cd GA-MonoSSC/
$ pip install -r requirements.txt

💡Note

Change L140 in depth_anything/metric_depth/zoedepth/models/base_models/dpt_dinov2/dpt.py to

self.pretrained = torch.hub.load('facebookresearch/dinov2', 'dinov2_{:}14'.format(encoder), pretrained=False)

Then, download Depth-Anything pre-trained model and metric depth model checkpoints file to checkpoints/.

4. Install tbb:

$ conda install -c bioconda tbb=2020.2

5. Finally, install GA-MonoSSC:

$ pip install -e ./

💡Note

If you move the GA-MonoSSC dir to another place, you should run

pip cache purge

then run pip install -e ./ again.

Datasets

NYUv2

1. Download the NYUv2 dataset.

2. Create a folder to store NYUv2 preprocess data at /path/to/NYU/preprocess/folder.

3. Store paths in environment variables for faster access:

$ export NYU_PREPROCESS=/path/to/NYU/preprocess/folder
$ export NYU_ROOT=/path/to/NYU/depthbin 

💡Note

Recommend using

echo "export NYU_PREPROCESS=/path/to/NYU/preprocess/folder" >> ~/.bashrc

format command for future convenience.

4. Preprocess the data to generate labels at a lower scale, which are used to compute the ground truth relation matrices:

$ cd GA-MonoSSC/
$ python GA-MonoSSC/data/NYU/preprocess.py NYU_root=$NYU_ROOT NYU_preprocess_root=$NYU_PREPROCESS

Running GA-MonoSSC

Training

NYUv2

1. Create folders to store training logs at /path/to/NYU/logdir.

2. Store in an environment variable:

$ export NYU_LOG=/path/to/NYU/logdir

3. Train GA-MonoSSC using 2 GPUs with batch_size of 4 (2 item per GPU) on NYUv2:

$ cd GA-MonoSSC/
$ python GA-MonoSSC/scripts/train.py \
    dataset=NYU \
    NYU_root=$NYU_ROOT \
    NYU_preprocess_root=$NYU_PREPROCESS \
    logdir=$NYU_LOG \
    n_gpus=2 batch_size=4

Evaluating

NYUv2

To evaluate GA-MonoSSC on NYUv2 test set, type:

$ cd GA-MonoSSC/
$ python GA-MonoSSC/scripts/eval.py \
    dataset=NYU \
    NYU_root=$NYU_ROOT\
    NYU_preprocess_root=$NYU_PREPROCESS \
    n_gpus=1 batch_size=1

Inference

Please create folder /path/to/GA-MonoSSC/output to store the GA-MonoSSC outputs and store in environment variable:

export GA_MonoSSC_OUTPUT=/path/to/GA-MonoSSC/output

NYUv2

To generate the predictions on the NYUv2 test set, type:

$ cd GA-MonoSSC/
$ python GA-MonoSSC/scripts/generate_output.py \
    +output_path=$GA_MonoSSC_OUTPUT \
    dataset=NYU \
    NYU_root=$NYU_ROOT \
    NYU_preprocess_root=$NYU_PREPROCESS \
    n_gpus=1 batch_size=1

Visualization

You need to create a new Anaconda environment for visualization.

conda create -n mayavi_vis python=3.7 -y
conda activate mayavi_vis
pip install omegaconf hydra-core PyQt5 mayavi

If you meet some problem when installing mayavi, please refer to the following instructions:

• Official mayavi installation instruction

NYUv2

$ cd GA-MonoSSC/
$ python GA-MonoSSC/scripts/visualization/NYU_vis_pred.py +file=/path/to/output/file.pkl

Aknowledgement

This project is built based on ISO. Please refer to (https://github.com/hongxiaoy/ISO) for more documentations and details.

We would like to thank the creators, maintainers, and contributors of the MonoScene, NDC-Scene, ZoeDepth, Depth Anything for their invaluable work. Their dedication and open-source spirit have been instrumental in our development.

Citation

@article{yu2024monocular,
  title={Global-Aware Monocular Semantic Scene Completion with State Space Models},
  author={Shijie Li, Zhongyao Cheng, Rong Li, Shuai Li, Juergen Gall, Xun Xu, Xulei Yang},
  journal={ICCV},
  year={2025}
}