COB-GS: Clear Object Boundaries in 3DGS Segmentation Based on Boundary-Adaptive Gaussian Splitting

Accepted by CVPR 2025

Webpage | Paper | arXiv

This repository contains the official authors implementation associated with the paper "COB-GS: Clear Object Boundaries in 3DGS Segmentation Based on Boundary-Adaptive Gaussian Splitting". We further introduce how to complete 3DGS segmentation with only images and text prompts.

Environment Setup

To prepare the environment,

1. Clone this repository.

   git clone https://github.com/ZestfulJX/COB-GS.git
   

2. Follow 3DGS to install dependencies.

   conda env create --file environment.yml
   conda activate cobgs
   

   Please notice, that the diff-gaussian-rasterization module contained in this repository has integrated the mask training branch to implement Boundary-Adaptive Gaussian Splitting.

3. Install Grounded-SAM-2.

   We provide a stable sequence masks extraction method based on Grounded-SAM-2 in ./submodules/Grounded-SAM-2-utils.

   cd submodules
   git clone https://github.com/IDEA-Research/Grounded-SAM-2.git
   cd Grounded-SAM-2 
   cd checkpoints
   wget https://dl.fbaipublicfiles.com/segment_anything_2/072824/sam2_hiera_large.pt
   cd ..
   cd gdino_checkpoints
   wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha2/groundingdino_swinb_cogcoor.pth
   cd ..
   
   pip install -e .
   pip install --no-build-isolation -e grounding_dino
   
   cd ../..
   cp ./submodules/Grounded-SAM-2-utils/grounded_sam2_tracking_demo.py ./submodules/Grounded-SAM-2
   

Run COB-GS

We provide process.sh to easily implement the complete segmentation process, which only requires the image sequence of the scene and the text prompts of the segmented parts.

1. Train 3DGS

  python train.py -s "dataset/tandt/truck" -m "output/truck"  --images "images_4"

2. Extract masks based on text prompt

  python submodules/Grounded-SAM-2/grounded_sam2_stable_tracking.py --dataset "tnt" --output "output" --scene "truck" --text "The truck" --resolution 4

3. Run 3DGS segmentation

  python train.py -s "dataset/tandt/truck" -m "output/truck" --start_checkpoint "output/truck/chkpnt30000.pth" --include_mask --finetune_mask --text "The truck" --images "images_4" --N4views 14 --mask_signals_threshold 0.8

• --include_mask: Add mask to the render.
• --finetune_mask: Split the boundary Gaussian using mask gradient. Using only include_mask does not change the structure of the scene.
• --N4views: L images, additionally optimize L*N4views epochs.
• --mask_signals_threshold: Threshold of relative distance.

Multi-Object Iterative Segmentation

The repository now also supports the paper-style multi-object iterative pipeline without modifying submodules/diff-gaussian-rasterization. The multi-object mode expects a single grayscale PNG mask per frame under:

<source_path>/object_mask/<image_basename>.png

• background value must be 0
• the same object must use the same grayscale value across views
• different objects in the same frame should use different grayscale values
• the mask basename must exactly match the source image basename, for example frame_0001.jpg -> frame_0001.png

Example training command:

python train.py -s "dataset/tandt/truck" -m "output/truck" --start_checkpoint "output/truck/chkpnt30000.pth" --include_mask --finetune_mask --mask_mode multi_label --mask_root "dataset/tandt/truck/object_mask" --images "images_4" --N4views 14 --object_order area_desc

Example rendering command:

python render.py -m "output/truck" --include_mask --mask_mode multi_label --images "images_4" --N4views 14 -w

Additional options:

• --mask_mode {single_text,multi_label}: choose between the original single-object text workflow and multi-label iterative segmentation
• --mask_root: path to the multi-label mask directory, default is <source_path>/object_mask
• --target_labels: comma-separated raw grayscale labels to process, for example --target_labels 32,64,128
• --object_order {area_desc,area_asc,label_asc}: iteration order for multi-object segmentation

Multi-object outputs are saved under:

<model_path>/multi_object/

• metadata.json: processed labels, iteration order, threshold, and mask statistics
• checkpoints/after_label_<label>.pth: checkpoint after each object is committed
• final_multi_object.pth: final checkpoint containing per-Gaussian object_id and object_score
• renders/object_<label>/<train|test>/*.png: binary mask renders for each object

Noting the need for fair comparison, we provide masks obtained on the NVOS dataset based on points prompts. Under our project, just put them under the ./output folder and skip Extract masks based on text prompt. Finally different scenes are evaluated in eval/eval_NVOS.py

We provide code for measuring the visual quality of textures using CLIP-IQA, along with our visual results. but it is important to note that this is only an expedient solution. The reason is that acquiring the real textures of segmented targets is challenging. If you are interested, please stay tuned for our follow-up work.

TODO List

• [✅] Provide demo and more visualizations.
• [✅] Update iterative multi-object segmentation.
• [ ] Update efficient texture optimizations.

Citation

If you find this project helpful for your research, please consider citing the report and giving a ⭐.

Any questions are welcome for discussion.

@inproceedings{zhang2025cobgs,
  title     = {COB-GS: Clear Object Boundaries in 3DGS Segmentation Based on Boundary-Adaptive Gaussian Splitting},
  author    = {Zhang, Jiaxin and Jiang, Junjun and Chen, Youyu and Jiang, Kui and Liu, Xianming},
  booktitle = {CVPR},
  year      = {2025}
}