jitter in the result of mixed_reality.py

[TimmmYang]

When I tried mixed reality demo and add moustache on celeba model, I ran

torchrun --nproc_per_node=4 applications/mixed_reality.py --ckpt celeba --objects --label_path assets/objects/celeba/celeba_moustache.png --sigma 0.3 --opacity 1 --real_size 1024 --resolution 2048 --real_data_path path_to_my_video --no_flip_inference

where I use 4 Nvidia 3090 GPUs and set --resolution = 2048 because of the memory limitation. The result makes sense but has some weird jitter as:

propagated.mp4

I checked the frames of the result and the images are normal with moustache. How do I fix this? Is there any incorrect parameters of my running?

[wpeebles]

Hi, can you confirm that you get the same jittery video when using 1 gpu (nproc_per_node=1)? If you do, I think the problem is likely with the name of your video frames before calling prepare_data.py; for example, if your frames are named 0.png, 1.png, 2.png, …, 10.png, 11.png then the order will be wrong. In order to correct the problem, the filenames of the frames should be padded (e.g., 0001.png instead of 1.png). If you have your input video in mp4, mov, etc. format you can use our process_video.sh script which takes care of padding. Once the files have been renamed, you can delete the previous LMDB created and rerun prepare_data.py. Let me know if you still have issues!

[TimmmYang]

It works now. Thanks for your suggestions!

[wpeebles]

Great!