Animate Static Photos into Talking Videos with LivePortrait AI Compose Perfect Expressions Fast #150
FurkanGozukara
announced in
Tutorials
Animate Static Photos into Talking Videos with LivePortrait AI Compose Perfect Expressions Fast
#150
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Animate Static Photos into Talking Videos with LivePortrait AI Compose Perfect Expressions Fast
Full tutorial: https://www.youtube.com/watch?v=FPtpNrmuwXk
With V3 update video to video added. If you are looking for a way to 1-Click install LivePortrait open source 0-shot image to animation application on Windows, run it locally, this is the tutorial that you need. In this tutorial I am introducing you the state-of-the-art image-to-animation open source generator Live Portrait. Provide your static image, your driving video, and in mere seconds have an amazingly working animation. LivePortrait is extremely fast and also capable of keeping the input video facial expressions. It will blow your mind when you see it. Believe me.
🔗 LivePortrait Installers Scripts⤵️
🔗 Requirements Step by Step Tutorial⤵️
🔗 Cloud Massed Compute, RunPod & Kaggle Tutorial (Mac users can follow this tutorial)⤵️
🔗 Official LivePortrait GitHub Repository⤵️
🔗 SECourses Discord Channel to Get Full Support⤵️
🔗 Paper of LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control⤵️
00:00:00 Introduction to the state-of-the-art image to animation open source application LivePortrait
00:02:20 How to download and install LivePortrait Gradio application on your computer
00:03:27 What are the requirements for LivePortrait application and how to install them
00:04:07 How to verify you have accurately installed requirements or not
00:05:02 How to verify installation was completed accurately and how to save installation logs
00:05:37 How to start the LivePortrait application after installation has been completed
00:05:57 The amazing extra materials I have shared such as portrait images, driving video and rendered videos
00:07:28 How to use LivePortrait application
00:08:06 How much VRAM LivePortrait uses when generating 73 seconds long animation video
00:08:33 Animating first image
00:08:50 How to monitor the status of animation process
00:10:10 First animation video is rendered
00:10:24 What is the resolution of the rendered animation videos
00:10:45 What is the original output resolution of the LivePortrait
00:11:27 Which improvements and new features I have coded top of the official demo app
00:11:51 Where the generated animated videos are saved by default
00:12:35 The effect of Relative Motion option
00:13:41 The effect of Do Crop option
00:14:17 The effect of Paste Back option
00:15:01 The effect of Target Eyelid Open Ratio option
00:17:02 How to join SECourses Discord channel
LivePortrait: an innovative framework for animating static portrait images to create realistic and expressive videos. The method aims to balance computational efficiency, generalization ability, and precise controllability.
Key features of LivePortrait:
It builds upon and extends implicit-keypoint-based methods rather than using diffusion-based approaches.
The model is trained in two stages:
Stage I: Base model training with enhancements like high-quality data curation, mixed image-video training, upgraded network architecture, scalable motion transformation, and landmark-guided optimization.
Stage II: Training of stitching and retargeting modules for improved controllability.
The framework introduces three key modules:
Stitching module: Allows seamless integration of animated portraits back into the original image space.
Eyes retargeting module: Enables precise control over eye movements and expressions.
Lip retargeting module: Provides fine-grained control over lip movements.
LivePortrait achieves impressive generation speed, producing animations in just 12.8ms on an RTX 4090 GPU using PyTorch.
The model outperforms many existing methods, including heavy diffusion-based approaches, in terms of generation quality and motion accuracy.
Key contributions:
Development of a solid implicit-keypoint-based video-driven portrait animation framework that significantly enhances generation quality and generalization ability.
Design of advanced stitching and retargeting modules for better controllability, with negligible computational overhead.
Extensive experiments demonstrating the efficacy of the framework in both self-reenactment and cross-reenactment scenarios.
The paper also discusses potential applications of LivePortrait in video conferencing, social media, entertainment, and audio-driven character animations. The authors acknowledge some limitations, such as difficulties with large pose variations in cross-reenactment scenarios and potential jitter with significant shoulder movements.
Noting the potential misuse of portrait animation technologies for deepfakes and the need for responsible use practices. The authors mention that current visual artifacts in synthesized results could aid in deepfake detection.
#ImageToAnimation #AIAnimation #PortraitAnimation
Video Transcription
00:00:00 Hello everyone. In this video, I am going to introduce you to the state-of-the-art image-to-animation
00:00:07 video generator open source application, LivePortrait. LivePortrait is remarkably capable of transferring expressions from input driving video into
00:00:17 animation with blazing speed. It can nearly mimic all of the expressions like this from your input video into your
00:00:27 animation video with extreme efficiency and accuracy. It is just mind-blowing.
00:00:31 You will see it. Generating 30 second video takes as low as 1 minute on RTX 3090 GPU
00:00:41 as low as only 4 gigabyte VRAM GPU. Just mind-blowing.
00:00:46 I have already prepared one-click installers for Windows, for MassedCompute, for RunPod, and even for a free Kaggle account.
00:00:57 So don't worry. If you don't have a powerful GPU, you will be able to use this amazing application on
00:01:05 a free Kaggle account as well.
00:01:07 And I am making some expressions for you to demonstrate the capability of this model right now.
00:01:13 So how we are going to generate such amazing image-to-animation videos, we are going to use LivePortrait advanced Gradio application.
00:01:23 This is an application that I have improved from the official repository. It has extra features.
00:01:29 In this tutorial video, I am going to show you how to one-click install this application
00:01:36 and use it on your computer with full details with all of the features.
00:01:41 So this will be a local installation tutorial on Windows computer.
00:01:46 However, I also have already prepared the cloud installers for MassedCompute, for Kaggle, for RunPod.
00:01:55 You can follow their instructions and use them already.
00:01:59 But after this video, there will be also another video for how to install and use on cloud.
00:02:05 Still, you need to watch this tutorial to learn how to use this amazing application
00:02:10 because on cloud tutorial, I will just show how to install and start running it.
00:02:15 So this tutorial is mandatory to learn how to use this amazing application.
00:02:20 So the scripts to install and start using LivePortrait on our computer and also on cloud
00:02:26 services are shared in this amazingly organized post with extra materials.
00:02:32 The link of this post will be in the description of the video.
00:02:36 Go to the very bottom of this post and you will see the attachments here.
00:02:41 You need to download LivePortrait version 2 zip file.
00:02:45 It can be a bigger version as well when you are watching this tutorial.
00:02:50 After download has been completed, move it into the disk where you want to install.
00:02:55 I am going to install into my R drive.
00:02:58 Do not install it into your downloads folder, into your users folder or into your cloud
00:03:04 folders like OneDrive and make sure that your folder path does not have space character.
00:03:11 So this is a good folder path. You see R drive LivePortrait version 2.
00:03:16 Then all you need to do is double click and start Windows install.bat file.
00:03:21 It will install everything automatically for you, including downloading the necessary models.
00:03:27 So what are the requirements for installers and this application to work on Windows?
00:03:32 You need to have Python 3.10.x version. I suggest 3.10.11, C++ tools, CUDA 11.8 and FFmpeg and Git.
00:03:44 If you don't know how to install these, I have an excellent tutorial here. Just click this link.
00:03:49 This is a 34 minutes excellent tutorial.
00:03:51 You can see all of the chapters of the tutorial and look at the parts that you need help with.
00:03:58 Install everything exactly as shown in this tutorial and you will not have any issues
00:04:03 with any AI application in future and also in this one.
00:04:07 So how you can verify your installations, open a CMD window like this and type Python.
00:04:13 You should get your Python version like this. Type FFmpeg like this, and you should get
00:04:17 a version like this. Type Git like this, and you should get a version like this. To verify
00:04:23 the CUDA installation type nvcc --version like this, and you should get the CUDA version like this.
00:04:32 Unfortunately, there is no easy way to check the C++ tools.
00:04:36 So watch this tutorial to install all of these requirements.
00:04:40 These requirements are the pillars of the open source AI applications.
00:04:44 So once you install them accurately, all of the AI applications will work flawlessly.
00:04:50 You see the installer is downloading the necessary models right now.
00:04:54 This is a very advanced script that I have prepared to make the installation flawlessly easy for you.
00:05:02 Okay, so the installation has been completed. Before closing this window,
00:05:08 just scroll to the top and verify everything is installed accurately. Moreover, you should save your installation logs.
00:05:16 So right-click here, edit, select all, right-click here, edit, copy, then paste it into a text editor and save it.
00:05:25 So if there be any errors, you can send me the error logs.
00:05:29 So I can find out the errors and fix them in the future if there be any errors.
00:05:35 Okay, so hit any key to close it.
00:05:37 Now after installation has been completed, all you need to do is double-click Windows
00:05:42 Start_app.bat file, and it will start the application automatically for you. You see the app is getting started.
00:05:50 This is a super ultra-fast application and the application interface has been loaded. It is so easy to use.
00:05:57 I also have shared amazing material for you.
00:06:02 You see I have 59 demo portrait images.zip file, click it to download from attachments.
00:06:08 I have prepared example driving video, click it to download from attachments.
00:06:13 And I have 59 demo portrait fully rendered videos. When you click here, it will download the zip file.
00:06:20 So what is inside this zip file?
00:06:23 Inside that zip file, you will see the 59 rendered images and their videos.
00:06:30 So why this is useful, this is useful that you can see what kind of images are working
00:06:36 better than the others. This is extremely useful. Let me show you one of them.
00:06:42 So you see, this is the full version. And when you open standalone version, it is like this.
00:06:50 So you can use these resources to see which faces are working best. Okay, let's go back to our downloads.
00:06:57 And you see we have the downloaded files here.
00:07:00 Let's return back to our folder and extract them here, right-click extract.
00:07:05 So we got the example driving video and also we got the demo portrait images.
00:07:11 You see these images, you can use all of them.
00:07:14 These are AI-generated images. I have used the SwarmUI to generate them.
00:07:19 If you don't know SwarmUI, I have made an amazing 90 minutes tutorial.
00:07:24 So you see it is on our channel, you can watch it, learn it and use it.
00:07:28 Okay, let's return back to our application. So select your source portrait. Let's go to our downloaded portrait images.
00:07:34 They are all here. Okay, let's use this face
00:07:38 for example. You see the face is loaded, then select your driving video.
00:07:42 I am sharing my example recorded video for you. Let me show you.
00:07:50 You can use simply any video. It is doing automatic cropping, relative motion and paste back and
00:07:57 what these options do I will show you. So this is a one minute 14 seconds video.
00:08:03 And you may be wondering how much VRAM it will use.
00:08:06 So to calculate the VRAM usage, I am going to use nvitop, but to be able to accurately
00:08:12 determine it, let's close the application. Okay. So when the application is closed, we are using 4.8 gigabyte VRAM.
00:08:20 So I'm starting the application again.
00:08:23 Let's load the image and the video again, make sure that both of them are uploaded accurately.
00:08:28 When you upload an image, if it doesn't recognize the face, you will get an error message here.
00:08:33 Then I am going to use default values and hit animate button here. So you will see the processing here.
00:08:40 The animated video will appear here and the full version will appear here.
00:08:44 Full version means that it includes the driving video and the input image as well.
00:08:50 And when you open the CMD window, the command line interface where the application started,
00:08:55 you can see the progress here.
00:08:57 So you see this 73 seconds long video is only going to take around 4 minutes of processing.
00:09:06 This is just mind-blowing, believe me. I have used the other image-to-animation, open source applications, and none of it is
00:09:15 fast as this one. We can also see the VRAM usage.
00:09:19 So it is using lesser than 4 gigabytes of VRAM.
00:09:22 If you have a 4 gigabyte GPU, it should work, but don't worry because I already have
00:09:28 Kaggle notebook, which works on a free Kaggle account amazingly fast.
00:09:32 I have already MassedCompute installer, and I also have RunPod installer.
00:09:38 The tutorials of these cloud services will be in a separate video.
00:09:42 It should be published very soon after this video, so you should be able to find them as well.
00:09:48 And when you render a shorter video, it takes much, much less duration.
00:09:52 It is like 1 second of video processing to 2 seconds of time.
00:09:58 So let's say when you process a 10 second video, it is taking around 20 seconds processing
00:10:04 time on RTX 3090 GPU. If you have a 4090, it is almost like real time.
00:10:10 Okay, so the video processing has been completed. We have the output video here. Let me play it. Hello everyone.
00:10:17 In this video, I am going to introduce you to the state-of-the-art image-to-animation.
00:10:24 Since we use paste back, the video resolution will be equal to my input image resolution,
00:10:31 which is currently 1280 pixels to 1280 pixels.
00:10:37 Because all of the portrait images that I provide you are 1280 pixels to 1280 pixels.
00:10:45 However, the original output resolution of the model is 512 to 512, which I will show you.
00:10:54 And there is also this stitched video. Let me also play it. Hello everyone.
00:10:58 In this video, I am going to introduce you to the state-of-the-art image-to-animation.
00:10:59 Since we use paste back, the video resolution will be equal to my input image resolution,
00:11:01 to the state-of-the-art image-to-animation. One minute on RTX 3090 GPU as low as only 4 gigabytes VRAM GPU just mind-blowing.
00:11:12 As you see, it is super high quality. It is so clean and it is ultra-fast.
00:11:18 This model is the very best model right now to turn static images into animated images.
00:11:26 And you will love it. So this Gradio application is based on the official repository. However, I have made some improvements
00:11:34 such as Target Eye Lip Open Ratio, this doesn't exist in the original repository. Moreover, the
00:11:40 video FPS issue is fixed and the output by default has the audio. And I also fixed how the
00:11:49 generated videos are saved. When you click this open outputs folder, it will open the folder where
00:11:56 every generated video is saved. You see, they will be saved inside animations folder inside the main
00:12:04 LivePortrait folder where you have installed. You see, this was our main installation folder.
00:12:10 This was the repository installed. And inside here, the animations will be the folder where
00:12:16 all of the images will be saved and they will be saved with a numbering. So when you generate
00:12:22 multiple videos with the same input image, they will not overwrite the previous one, they will get
00:12:29 a new number. Now I will show you what each option does one by one. Okay, so the first option is
00:12:37 Relative Motion. And let's see what does this option do. Hello, everyone. In this video, I am
00:12:44 going to introduce you to the state-of-the-art image-to-animation video generator. So you see
00:12:51 the head movement has been changed. This was with the Relative Motion so you can see the difference
00:12:58 of them. I will open both of them at the same time. Okay, so on the left, we have the original
00:13:04 output and on the right, we have the output of Relative Motion is off. So let's see the difference.
00:13:13 Hello, everyone. In this video, I am going to introduce you to the state-of-the-art
00:13:18 image-to-animation video generator open source application LivePortrait. LivePortrait is
00:13:25 remarkably capable of transferring expressions from input driving video into animation with
00:13:32 blazing speed. So this is the impact of relative motion. Try it and see if you like it more or not.
00:13:40 So when I turn off Do Crop, this is the result I get. Why? I get this result because we also have
00:13:48 paste back option enabled. So when we disable Do Crop, probably we need to also disable Paste Back.
00:13:56 I am going to regenerate this video to see the difference. Currently it is unusable like this as
00:14:02 you are seeing. Alright, so after I have disabled Do Crop and Paste Back, this is the result. So
00:14:09 let's see the resolution of this video. It is only 512. So give it a try if you like it or not.
00:14:17 And then there is paste back option. So what paste back option does is by default the model
00:14:24 generates 512 headshot like this, let me show you. So paste back option, paste the generated
00:14:32 video into the full original image. Therefore, with that way, we are able to generate same
00:14:39 resolution as our input image. Let me download this and show you. So you see the original
00:14:44 generated video is only 512 to 512. However, with paste back option, it automatically scales
00:14:52 and pastes the generated portrait head into the input image and generates a bigger resolution
00:14:59 video. This is how it works. And finally, the Target Eye Lip Open Ratio. So what this option
00:15:06 does when you go to the very bottom, you will see target eyes open ratio and target lip open ratio.
00:15:14 Currently, I have used 50% and 70%. So let's make this 50% like this and 70% like this.
00:15:22 And after doing that, when you click retargeting, you will see that it will make the eyes more open
00:15:29 and the mouth more open like this. So let's compare this video with the original default output video.
00:15:38 Now on the left, we see the default output and on the right, we see the Target Eye Lip Open Ratio,
00:15:46 50% eyes more open and 70% lips more open. Let's play and see the results. Hello, everyone. In this
00:15:54 video, I am going to introduce you to the state-of-the-art image-to-animation, video generator,
00:16:01 open source application, LivePortrait. LivePortrait is remarkably capable of transferring
00:16:07 expressions from input driving video into animation with blazing speed. It can nearly
00:16:13 mimic all of the expressions like this. So you see this is the difference. When you enable
00:16:20 Target Eye Lip Open Ratio, the movement of the head eyes and the lip is gone. It is not exist
00:16:29 anymore. So this is a limitation currently. I have reported this to the developers of the
00:16:34 LivePortrait. And if this can be fixed, hopefully it will be also fixed with my application when
00:16:41 you're watching this video. And this may be useful for some people in some cases. In the original
00:16:47 repository, this is not by default supported. So I have coded this myself in my special application.
00:16:54 This is why supporting me is extremely important and useful for you. I hope that you have enjoyed.
00:17:01 Please like, subscribe and share our video. Also, please join our Discord channel. When you click
00:17:08 this link, you will get to our Discord channel. Our Discord channel is for all members. You see,
00:17:13 we have over 7000 members. You don't have to be my Patreon supporter. Just by typing Google
00:17:19 SECourses Discord, you will get to this page. Also, this is super important. Please click this
00:17:25 link. This is our main GitHub repository. Please Watch it, Fork it, Star it and Sponsor it if you can.
00:17:34 If you sponsor, I appreciate that very much. The link of this page will be in the description of the video.
00:17:40 And you see this is the official repository link. Hopefully see you in another amazing tutorial
00:17:44 video. I am working on Instant ID and it will make you super excited and surprised because with zero
00:17:52 shot it is working amazing. I have coded a new way of using the Instant ID that no one has ever done
00:17:58 yet. So you will like it and I will also hopefully publish cloud tutorials. So if you are going to
00:18:03 use this amazing application on the cloud, you will like it. Thank you so much for watching. Hopefully see you later.
Beta Was this translation helpful? Give feedback.
All reactions