This is a simple pipeline to edit the demo video by inpainting. For the Heygen sample video, the pipeline works as follows
-
Extract frames and run get the
person mask
, by running a depth estimator (ZoeDepth) and then seperate the foreground from backgroud. -
Run face detector on each frame to get the
face mask
. The difference ofperson mask
minusface mask
is the region that needs inpainting. -
Run stable diffusion inpainting (with controlnet) on a bunch of fashion prompts to fill the new clothes for the person.
-
(Manual) Pick the good generated key images.
-
Use ebsynth to synthesize the frames, and concat the frames into the final mp4.
out.mp4
Example stable diffusion inpainting results for keyframes:
pip install -r requirements.txt
# zoe depth
git clone https://github.com/isl-org/ZoeDepth
# ebsynth
git clone https://github.com/SingleZombie/ebsynth.git
python install.py
# optional: face parsing, for getting better face masks for inpainting
git clone https://github.com/zllrunning/face-parsing.PyTorch.git
python gdown.py
# prepare data
python prepare_data.py
# run the pipeline (-ri: inpainting, -re: ebsynth, -rf: ffmpeg)
python pipeline.py -ri -re -rf
- The example output is generated by the ebsynth app on Mac.
The programmable version of
ebsynth
used in this repo doesn't seem to support the advanced features such as de-clicker, which leads to inferior results if you directly run thepipeline.py
. See this upper body video (with audio) as the raw output of the pipeline.
output_upper_body.mp4
-
The identity-preserving feature is naturally achieved by fixing the face unchanged using inpainting masks, so we can't edit the faces yet. To have more editing flexibility, I tried Instruct-Pix2Pix to achieve wilder editing but the output quality isn't consistent.
-
Having issue generating good hands for Half body.