GenHeld: Generating and Editing HandHeld Objects

Paper | Project Page

Installation

git clone --recursive https://github.com/ChaerinMin/GenHeld.git

This code was implemented under Ubuntu 22.04, NVIDIA RTX 4060 Ti, cuda 11.8, and gcc 10.5.0.

1. Create a conda environment

   conda create -n genheld python=3.8.18
   conda activate genheld
   pip install torch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 --index-url https://download.pytorch.org/whl/cu118

2. Install packages

   conda install -c fvcore -c iopath -c conda-forge fvcore iopath
   pip install -r requirements.txt
   pip install -U openmim
   mim install mmcv-full==1.7.2

3. V-HACD

   Install v-hacd and change /home/cmin5/v-hacd/app/build/TestVHACD to your own path at configs/metric/metric.yaml.

4. Blender

   Visit Blender, download, and unzip. We used version 4.0.2. If your file system is like below,

   <home directory>
   |-- blender
   |   |-- lib
   |   |-- license
   |   |-- blender
   |   |-- blender.desktop

   add <home directory>/blender to $PATH.

Datasets

Required data

Download all files and folders from https://drive.google.com/drive/folders/1klKIZ7CsX0gnNH0c741VU88YeI5mOgJZ?usp=sharing and place them under assets/ The file structure should look like below.

GenHeld
|-- assets
|   |-- contact_zones.json
|   |-- lama
|   |   |-- big-lama
|   |   |   |-- ...
|   |   |-- default.yaml
|   |-- sam
|   |   | -- sam_vit_h_4b8939.pth
|   |-- ...
|-- main.py
|-- ...

For reference, find MANO, NIMBLE, SMPLX, SAM, LAMA, EgoHOS and HiFiHR. We adopted the contact_zones.pkl from Obman, and preprocessed SMPLX arms from HARP.

Hands

FreiHAND

Run the following code to download FreiHAND.

cd <your storage>
mkdir FreiHAND
cd FreiHAND
wget https://lmb.informatik.uni-freiburg.de/data/freihand/FreiHAND_pub_v2.zip  # 3.7GB
unzip FreiHAND_pub_v2.zip  # 4.6GB
rm FreiHAND_pub_v2.zip
cd <GenHeld>
ln -s <abs path to FreiHAND> data/

Then, your file system should look like below.

GenHeld
|-- data
|   |-- FreiHAND
|   |   |-- evaluation
|   |   |   |-- ...
|   |   |-- ...
|-- main.py
|-- ...

Objects

YCB

Run the following code to download YCB.

python data/download_ycb.py --download_dir <your storage>
ln -s <abs path to YCB> data/

Then, your file system should look like below.

GenHeld
|-- data
|   |-- FreiHAND
|   |   |-- evaluation
|   |   |-- ...
|   |-- YCB
|   |   |-- 001_chips_can
|   |   |-- ...
|-- main.py
|-- ...

Objaverse

Download the preprocessed data from https://drive.google.com/drive/folders/1xVGupsuKQnV8GSOnLJepU0FUBPqsv5zy?usp=sharing and make symbolic link if you want, ln -s <abs path to Objaverse> data/

Then, your file system should look like below.

GenHeld
|-- data
|   |-- FreiHAND
|   |   |-- evaluation
|   |   |-- ...
|   |-- YCB
|   |   |-- 001_chips_can
|   |   |-- ...
|   |-- Objaverse
|   |   |-- banana
|   |   |-- ...
|-- main.py
|-- ...

GenHeld 3D

Object Selection Network

To use the pretrained Object Selection network, make sure the path to the checkpoint is same as ckpt_path under selector in the configs/config.yaml file. Then, proceed to the Object Fitting.

If you want to train by yourself, download DexYCB and place it under data/ with name DexYCB, and run

python main.py selector.mode=train selector.ckpt_path=""

Object Fitting

We provide the pre-fitted results at https://drive.google.com/drive/folders/1ADdnR9WYFwQ5KUn0YdXnf0jpV0GhceVd?usp=sharing Make a directory named outputs/prefit and put the files under it.

GenHeld
|-- outputs
|   |-- prefit
|   |   |-- checkpoints
|   |   |-- ...
|-- main.py

To run the evaluation,

python main.py object_dataset=ycb select_object=category selector.ckpt_path=assets/selector_checkpoints/epoch=151-step=456.ckpt optimizer.mode=evaluate optimizer.resume_dir=outputs/prefit/

Object fitting by yourself

You might want to run the Object Selection Network and fit the object by yourself. To do this,

python main.py optimizer.mode=optimize

GenHeld 2D

We hinge our GenHeld3D upon DiffEditor to achieve GenHeld2D. Install DiffEditor and open the gradio (python app.py).

In the Object pasting tab, put the input image as Original image and put the xx_warped.png as Reference image. xx_warped.png will be under either outputs/xx/xx/evaluations (if you only run evaluation) or outputs/xx/xx/results. Make a box around the object and run.

Acknowledgement

This work was supported by NASA grant #80NSSC23M0075, and NSF CAREER grant #2143576.

Citation

@misc{min2024genheld,
    title={GenHeld: Generating and Editing Handheld Objects}, 
    author={Chaerin Min and Srinath Sridhar},
    year={2024},
    eprint={2406.05059},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}