In this work, we present Vlogger, a generic AI system for generating a minute-level video blog (i.e., vlog) of user descriptions. Different from short videos with a few seconds, vlog often contains a complex storyline with diversified scenes, which is challenging for most existing video generation approaches. To break through this bottleneck, our Vlogger smartly leverages Large Language Model (LLM) as Director and decomposes a long video generation task of vlog into four key stages, where we invoke various foundation models to play the critical roles of vlog professionals, including (1) Script, (2) Actor, (3) ShowMaker, and (4) Voicer. With such a design of mimicking human beings, our Vlogger can generate vlogs through explainable cooperation of top-down planning and bottom-up shooting. Moreover, we introduce a novel video diffusion model, ShowMaker, which serves as a videographer in our Vlogger for generating the video snippet of each shooting scene. By incorporating Script and Actor attentively as textual and visual prompts, it can effectively enhance spatial-temporal coherence in the snippet. Besides, we design a concise mixed training paradigm for ShowMaker, boosting its capacity for both T2V generation and prediction. Finally, the extensive experiments show that our method achieves state-of-the-art performance on zero-shot T2V generation and prediction tasks. More importantly, Vlogger can generate over 5-minute vlogs from open-world descriptions, without loss of video coherence on script and actor.
Teddy_Travel_low.mp4
conda create -n vlogger python==3.10.11
conda activate vlogger
pip install -r requirements.txtOur model is based on Stable diffusion v1.4, you may download Stable Diffusion v1-4 and OpenCLIP-ViT-H-14 to the director of pretrained
.
Download our model(ShowMaker) checkpoint (from google drive or hugging face) and save to the directory of pretrained
Now under ./pretrained, you should be able to see the following:
├── pretrained
│ ├── ShowMaker.pt
│ ├── stable-diffusion-v1-4
│ ├── OpenCLIP-ViT-H-14
│ │ ├── ...
└── └── ├── ...
├── ...
Run the following command to get script, actors and protagonist:
python sample_scripts/vlog_write_script.py-
The generated scripts will be saved in
results/vlog/$your_story_dir/script. -
The generated reference images will be saved in
results/vlog/$your_story_dir/img. -
⚠️ Enter your openai key in the 7th line of the filevlogger/planning_utils/gpt4_utils.py
Run the following command to get the vlog:
python sample_scripts/vlog_read_script_sample.py- The generated scripts will be saved in
results/vlog/$your_story_dir/video.
Run the following command to get the (T+I)2V results:
python sample_scripts/with_mask_sample.py- The generated video will be saved in
results/mask_no_ref.
Run the following command to get the (T+I+Ref)2V results:
python sample_scripts/with_mask_ref_sample.py- The generated video will be saved in
results/mask_ref.
You may modify configs/with_mask_sample.yaml to change the (T+I)2V conditions and modify configs/with_mask_ref_sample.yaml to change the (T+I+Ref)2V conditions.
For example:
-
ckptis used to specify a model checkpoint. -
text_promptis used to describe the content of the video. -
input_pathis used to specify the path to the image. -
ref_pathis used to specify the path to the reference image. -
save_pathis used to specify the path to the generated video.
| Reference Image | Output Video |
Scene Reference |
Fireworks explode over the pyramids. |
Scene Reference |
The Great Wall burning with raging fire. |
Object Reference |
A cat is running on the beach. |
| Input Image | Output Video |
 |
Underwater environment cosmetic bottles. |
 |
A big drop of water falls on a rose petal. |
 |
A fish swims past an oriental woman. |
 |
Cinematic photograph. View of piloting aaero. |
 |
Planet hits earth. |
| Output Video | |
A deer looks at the sunset behind him. |
A duck is teaching math to another duck. |
Bezos explores tropical rainforest. |
Light blue water lapping on the beach. |
@article{zhuang2024vlogger,
title={Vlogger: Make Your Dream A Vlog},
author={Zhuang, Shaobin and Li, Kunchang and Chen, Xinyuan and Wang, Yaohui and Liu, Ziwei and Qiao, Yu and Wang, Yali},
journal={arXiv preprint arXiv:2401.09414},
year={2024}
}@article{chen2023seine,
title={SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction},
author={Chen, Xinyuan and Wang, Yaohui and Zhang, Lingjun and Zhuang, Shaobin and Ma, Xin and Yu, Jiashuo and Wang, Yali and Lin, Dahua and Qiao, Yu and Liu, Ziwei},
journal={arXiv preprint arXiv:2310.20700},
year={2023}
}@article{wang2023lavie,
title={LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models},
author={Wang, Yaohui and Chen, Xinyuan and Ma, Xin and Zhou, Shangchen and Huang, Ziqi and Wang, Yi and Yang, Ceyuan and He, Yinan and Yu, Jiashuo and Yang, Peiqing and others},
journal={arXiv preprint arXiv:2309.15103},
year={2023}
}We disclaim responsibility for user-generated content. The model was not trained to realistically represent people or events, so using it to generate such content is beyond the model's capabilities. It is prohibited for pornographic, violent and bloody content generation, and to generate content that is demeaning or harmful to people or their environment, culture, religion, etc. Users are solely liable for their actions. The project contributors are not legally affiliated with, nor accountable for users' behaviors. Use the generative model responsibly, adhering to ethical and legal standards.
Shaobin Zhuang: zhuangshaobin@pjlab.org.cn, Kunchang Li: likunchang@pjlab.org.cn
Xinyuan Chen: chenxinyuan@pjlab.org.cn, Yaohui Wang: wangyaohui@pjlab.org.cn
The code is built upon SEINE, LaVie, diffusers and Stable Diffusion, we thank all the contributors for open-sourcing.
The code is licensed under Apache-2.0, model weights are fully open for academic research and also allow free commercial usage. To apply for a commercial license, please contact zhuangshaobin@pjlab.org.cn.