title | emoji | colorFrom | colorTo | sdk | sdk_version | app_file | pinned |
---|---|---|---|---|---|---|---|
Seine |
😊 |
pink |
pink |
gradio |
4.3.0 |
app.py |
false |
This repository is the official implementation of SEINE.
SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction
conda env create -f env.yaml
conda activate seine
Download our model checkpoint from Google Drive and save to directory of pre-trained
Our model is based on Stable diffusion v1.4, you may download Stable Diffusion v1-4 to the director of pre-trained
Now under ./pretrained
, you should be able to see the following:
├── pretrained_models
│ ├── seine.pt
│ ├── stable-diffusion-v1-4
│ │ ├── ...
└── └── ├── ...
├── ...
python sample_scripts/with_mask_sample.py --config configs/sample_i2v.yaml
The generated video will be saved in ./results/i2v
.
python sample_scripts/with_mask_sample.py --config configs/sample_transition.yaml
The generated video will be saved in ./results/transition
.
You can modify ./configs/sample_mask.yaml
to change the generation conditions.
For example,
ckpt
is used to specify a model checkpoint.
text_prompt
is used to describe the content of the video.
input_path
is used to specify the path to the image.
@article{chen2023seine,
title={SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction},
author={Chen, Xinyuan and Wang, Yaohui and Zhang, Lingjun and Zhuang, Shaobin and Ma, Xin and Yu, Jiashuo and Wang, Yali and Lin, Dahua and Qiao, Yu and Liu, Ziwei},
journal={arXiv preprint arXiv:2310.20700},
year={2023}
}