This is the official repository of HOSNeRF Project page | arXiv | Video
Jia-Wei Liu, Yan-Pei Cao, Tianyuan Yang, Zhongcong Xu, Jussi Keppo, Ying Shan, Xiaohu Qie, Mike Zheng Shou
TL;DR: A novel 360° free-viewpoint rendering method that reconstructs neural radiance fields for dynamic human-object-scene from a single monocular in-the-wild video.
HOSNeRF can render 360° free-viewpoint videos from a single monocular in-the-wild video.
HOSNeRF Framework.
-
[2023.12.11] We release the HOSNeRF codebase!
-
[2023.08.16] We release the HOSNeRF dataset!
-
[2023.08.12] HOSNeRF got accepted by ICCV 2023!
-
[2023.04.24] We release the arXiv paper!
git clone https://github.com/TencentARC/HOSNeRF.git
cd HOSNeRF
pip install -r requirements.txt
Download the gender neutral SMPL model from here, and unpack mpips_smplify_public_v2.zip.
Copy the smpl model.
SMPL_DIR=/path/to/smpl
MODEL_DIR=$SMPL_DIR/smplify_public/code/models
cp $MODEL_DIR/basicModel_neutral_lbs_10_207_0_v1.0.0.pkl third_parties/smpl/models of 2nd_State_Conditional_Human-Object and 3rd_Complete_HOSNeRF
Follow this page to remove Chumpy objects from the SMPL model.
We release the HOSNeRF dataset on link. HOSNeRF dataset consists of 6 real-world dynamic human-object-scene sequences: Backpack, Tennis, Suitcase, Playground, Dance, Lounge. Please run the optical flow estimation method using RAFT to get the optical flows of each scene.
Stage 1: Train the state-conditional background model.
$ cd 1st_State-Conditional_Scene
$ CUDA_VISIBLE_DEVICES=0,1,2,3 python run.py --ginc configs/state_mipnerf360/Backpack.gin --scene Backpack --logbase 'path to logbase'Stage 2: Train the state-conditional dynamic human-object model.
(Please also change the datadir in configs/default.yaml and core/data/dataset_args.py)
$ cd 2nd_State_Conditional_Human-Object
$ CUDA_VISIBLE_DEVICES=0,1,2,3 python run.py --ginc configs/human-object/Backpack.gin --scene Backpack --logbase 'path to logbase' --cfg configs/human_nerf/wild/monocular/adventure.yaml --seed 777Stage 3: Train the complete HOSNeRF model using the trained background and human-object checkpoints.
(Please also change the datadir in configs/default.yaml and core/data/dataset_args.py)
$ cd 3rd_Complete_HOSNeRF
$ CUDA_VISIBLE_DEVICES=0,1,2,3 python run.py --ginc configs/HOSNeRF/Backpack.gin --scene Backpack --logbase 'path to logbase' --cfg configs/human_nerf/wild/monocular/adventure.yaml --seed 777We include the test codes in the model's test_step function. It will automatically run the test metrics (PSNR, SSIM, and LPIPS) for test images and all images after training.
Please change the freeview index in the configs/default.yaml to render the free-viewpoint videos of that timestep. It will automatically render free-viewpoint videos (the model's test_step function) after training.
It will automatically render the canonical human-object videos (the model's test_step function) after training.
To resume training or resume testing after training, please add --resume_training True for each training script.
We release the 6 HOSNeRF checkpoints on link for reference.
If you find our work helps, please cite our paper.
@inproceedings{liu2023hosnerf,
title={Hosnerf: Dynamic human-object-scene neural radiance fields from a single video},
author={Liu, Jia-Wei and Cao, Yan-Pei and Yang, Tianyuan and Xu, Zhongcong and Keppo, Jussi and Shan, Ying and Qie, Xiaohu and Shou, Mike Zheng},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
pages={18483--18494},
year={2023}
}This repo is maintained by Jiawei Liu. Questions and discussions are welcome via jiawei.liu@u.nus.edu.
This codebase is based on HumanNeRF and NeRF-Factory. The preprocessing code is based on NeuMan. Thanks for open-sourcing!
Copyright (c) 2023 Show Lab, National University of Singapore. All Rights Reserved. Licensed under the Apache License, Version 2.0 (see LICENSE for details)