OSTRA

OSTRA is a novel segmentation-then-reconstruction method for segmenting complex open objects in 3D point clouds. This method uses a Segment-Anything Model (SAM) to segment target objects and video object segmentation (VOS) technology to continuously track video frame segmentation targets. Our pipeline enables a complete segmentation process from videos to 3D cloud points and meshes in different level(semantic segmentation, instance segmentation and part segmentation).

You can check our detailed tutorials here!

💻Getting Started

Clone OSTRA

This project is tested under python3.9, cuda11.5 and pytorch1.11.0. An equivalent or higher version is recommended.

#Clone OSTRA
git clone --recursive https://github.com/ganlab/OSTRA.git
cd OSTRA

#Install dependencies:
pip install -r requirements.txt

Install Colmap

Our reconstruction process is based on Colmap. Please follow the instruction and install Colmap first.

Prepare Model

All these models are required for OSTRA: SAM: the default model is sam_vit_b_01ec64.pth DeAOT:the default model is R50_DeAOTL_PRE_YTB_DAV.pth XMem：the default model is XMem-s012.pth Grounding-DINO:the default model is groundingdino_swint_ogc.pth

WebUI

We developed WebUI that user can easily access.

python app.py --device cuda:0

📽️ Demo

Two samples of complex object segmentation:

📖Citation

Please considering cite our paper if you find this work useful!

@misc{xu2023stop,
      title={A One Stop 3D Target Reconstruction and multilevel Segmentation Method}, 
      author={Jiexiong Xu and Weikun Zhao and Zhiyan Tang and Xiangchao Gan},
      year={2023},
      eprint={2308.06974},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

👏 Acknowledgements

This work is based on Segment Anything, Track Anything, Segment and Track Anything, Colmap and Open3D.