Vector sketch animation generation with differentialable motion trajectories

Xinding Zhu, Xinye Yang, Shuyang Zheng, Zhexin Zhang, Fei Gao, Jing Huang, Jiazhou Chen

1. Install

git clone ...
cd DMTSketch
conda create -n diffanim python=3.7
conda activate diffanim
conda install cuda=11.6
pip install torch==1.13.1+cu116 torchvision==0.14.1+cu116 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu116
pip install -r requirements.txt
conda install -y numpy
git clone https://github.com/BachiLi/diffvg
conda install -y -c anaconda cmake
cd diffvg
git submodule update --init --recursive
python setup.py install
pip install git+https://github.com/openai/CLIP.git
cd ..
pip install ipywidgets icecream

2. Calculate the tracking information using the co-tracker

If other methods are used to obtain tracking information, this step can be skipped.

2.1 Install co-tracker

The specific command reference https://github.com/facebookresearch/co-tracker

git clone https://github.com/facebookresearch/co-tracker
cd co-tracker
conda create -n co-tracker python=3.12
conda activate co-tracker
conda install cuda=12.4
conda install pytorch==2.5.0 torchvision==0.20.0 torchaudio==2.5.0 pytorch-cuda=12.4 -c pytorch -c nvidia
pip install -e .
pip install matplotlib flow_vis tqdm tensorboard imageio imageio[ffmpeg]
mkdir -p checkpoints
cd checkpoints
wget https://huggingface.co/facebook/cotracker3/resolve/main/scaled_online.pth
wget https://huggingface.co/facebook/cotracker3/resolve/main/scaled_offline.pth
cd ../..

2.2 Calculate tracks

conda activate co-tracker
python calculate_tracks.py --video_path <VideoPath> --output_dir <outputPath>

for example

conda activate co-tracker
python calculate_tracks.py --video_path "data/input/mallard-water/mallard-water.mp4" --output_dir "data/input/mallard-water/"

3. Run Demo

conda activate diffanim
python run_para_clip_video.py --target_name <TargetName> --frame_num <FrameNumber> --order_num <PolynomialTermNumber> --num_paths <StrokeNumber>

for example

conda activate diffanim
python run_para_clip_video.py --target_name mallard-water --frame_num 60 --order_num 12 --num_paths 8

Other arguments:

• --fps_rate Defines the ratio of the output frame number to the input frame number, with the default value being 1. If you need to output a result with a higher frame rate, you can set it to a larger value.
• --save_interval Defines the interval for saving the results. The default is 100.
• --consistent_weight Defines the weight of the consistency loss function. The default is 2.
• --device Defines the driver number of the GPU. The default is 0.

More data can be downloaded at https://drive.google.com/drive/folders/1_a4ffzYWCqAi5uhx2VMogpFKCAY0Fjmq

Citation

@article{zhu2025vector,
  title={Vector sketch animation generation with differentialable motion trajectories},
  author={Zhu, Xinding and Yang, Xinye and Zheng, Shuyang and Zhang, Zhexin and Gao, Fei and Huang, Jing and Chen, Jiazhou},
  journal={arXiv preprint arXiv:2509.25857},
  year={2025}
}

Acknowledgements

Thank CLIPasso, CoTracker for wonderful works and publicly releasing their code.

License

Shield:

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.