The code is distributed under the MIT License.
pip install -r requirements.txt
Essential python libraries:
PyYAML==6.0.1
torch==2.0.1
torchgeometry==0.1.2
trimesh==3.23.5
numpy==1.23.1
We exploited this dataset for novel point motion prediction. Before downloading this dataset, one should check its official site and agree its term of use. Please see our extended ResField for how we processed this dataset.
We created this dataset with 4 sequences for novel point motion prediction. One can download our version here, or generate new synthetic data via utils/gen_syntheticdataset_3d.py
.
We exploited this dataset for temporal mesh alignment with guidance. Please see its official website for more information and downloading.
In our experiment, we choose 16 sequences from 4 subjects in the packed
sequences in the test split.
For each sequence, we first perform down-sampling by every 2 frames, and then select the first 30 frames as the new sequence to model. The employed sequences are present in our paper supp. mat.
We conducted an additional experiment in the supp. mat, and leveraged this github repo to produce the data. Running and extracting the simulated particles require some basic knowledge of Unity3D. Alternatively, one can download our extracted and processed data here.
The training, evaluation, and rendering scripts are in these train_*.py
files.
Besides the three baselines implemented in our modified ResField repo, one can run set up the dataset path, and run e.g.
python train_deformingthings4d.py --motion_model_name=affinefield4d
All options of motion_model_name
are listed in the MotionField
class in models/dpfs.py
.
The results are saved into the output
folder. One can modify the point renderer settings in render_points()
of utils/vis.py
. Details of the configurations are in train_deformingthings4d.py
.
Similarly, one can use the following to learn motion fields for the 4 synthetic sequences.
python train_synthetic.py --motion_model_name=affinefield4d --homo_loss_weight=0.1
Run the following one to learn the motion field in the simulated fluid field.
python train_particlesim.py --sequence=/doma_datasets/particlesim/particles_0000.npy --motion_model_name=transfield4d --start=100 --end=110
One can start with train_resynth.py
, e.g.
python train_resynth.py --subject=rp_aaron_posed_002 --seq=96_jerseyshort_hips --motion_model_name=affinefield4d
@inproceedings{DOMA,
title = {Degrees of Freedom Matter: Inferring Dynamics from Point Trajectories},
author = {Yan Zhang and Sergey Prokudin and Marko Mihajlovic and Qianli Ma and Siyu Tang},
booktitle = {Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
year = {2024}
}
@inproceedings{mihajlovic2024ResFields,
title={{ResFields}: Residual Neural Fields for Spatiotemporal Signals},
author={Mihajlovic, Marko and Prokudin, Sergey and Pollefeys, Marc and Tang, Siyu},
booktitle={International Conference on Learning Representations (ICLR)},
year={2024}
}
@inproceedings{prokudin2023dynamic,
title = {Dynamic Point Fields},
author = {Prokudin, Sergey and Ma, Qianli and Raafat, Maxime and Valentin, Julien and Tang, Siyu},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
pages = {7964--7976},
month = oct,
year = {2023}
}