Speed optimisation using multiple processes

[svleeuwen]

Hi,

I came across this post antiboredom/videogrep#3 were you highlight the fact that each VideoFileClip object creates an ffmpeg process.
I noticed that I didn't take care of this in my code as well so I changed it so it reuses the same VidoeFileClip.

Howerver, this has a drastic impact on the total rendering time. It takes a lot longer to render the video now.

Is it bad to just let MoviePy create multiple ffmpeg processes?
Is there another way to reduce rendering time? (I'm at 1.80 iters/sec now)

Regards,
Sander

[Zulko]

It's not bad to create multiple processes, it's just that it takes some place in the RAM so you can't have more than, say, 100 VideoFileClips open at the same time.

I'm not sure what you mean by reusing the same VideoFileClip, but the general idea is that when you have to access several scenes of the same video file at the same time (like, for embedding one scene of a movie inside another scene from the same movie, or putting two scenes side by side on the same screen) then it is better to create several VideoFileClips which will act as different entry points for the videofile.

[svleeuwen]

Allright, thanks.

I'm making a 3x3 grid of video's which can be made of the same video with a variable start_time. So you would suggest multiple VideoFileClips for this use case?

[Zulko]

Yes, definitely, this way each videofile clip is a different decoding process and reads only consecutive frames.

What happened in your code is that your process needed to access several frames at once at very different times, and the only way to do that is to restart the whole decoding process a lot. This is what was slow.

[svleeuwen]

Great, thanks a lot!

[sibebleuze]

Yes, definitely, this way each videofile clip is a different decoding process and reads only consecutive frames.

What happened in your code is that your process needed to access several frames at once at very different times, and the only way to do that is to restart the whole decoding process a lot. This is what was slow.

I'm just getting started with moviepy and I'm also experiencing low it/s like svleeuwen, something between 2 and 3 for me. What I am trying to do is to take a 1 hour long video and put some text on it with SubtitlesClip, as well as the sound that I'm getting out of a .ogg file, also 1 hour of sound here, this might be an mp3 or even mp4 file in future projects of mine. So if I understand you correctly, I would get faster processing if I load the video and audio files in different Clips, so for example make a VideoFileClip().subclip(0,1200) for the first 20 minutes and further subclips for the rest of the video? Or is it because the file is this big to begin with that it's this slow and there's not really anything I can do? What is your advice on this?

[sibebleuze]

Ok, so I tried reading in the video and audio in clips of 5 minutes (and some other lengths too, for comparison), as well as creating the added text in the same 5 minute clips, then adding them together with CompositeVideoClip (per 5 minutes) and concatenating with concatenate_videoclips and writing the result of this to an mp4 file. This had the effect described above, it created a lot of ffmpeg instances. However, the speed gain from this was not really what I expected, which might have something to do with only one of the ffmpeg instances really using a considerable amount of processing power (as measured by Windows Task Manager). I got about twice the it/s now, so about 5, but for a 1 hour video at 1200x1600 resolution and 30 fps, this is still gonna take around 7 hours to process the whole thing. Is this what you would expect @Zulko or is there still speed to be gained. I read on some other issues you guys were working on multithreading/multiprocessing for MoviePy, will that speed it up even more maybe? Or am I just expecting too much for this big of a video? I really don't know, as I said I just started and I have no idea how long these things take.