Effects Processing Issue

[peanutbutterandcrackers]

This morning I was creating a video for a pal when I noticed a few issues with the application of effects to a clip (an image, in my case) on the timeline.

1. It seems that OpenShot was only using one core of the Intel Core i5 8th Gen machine to apply the effect to the clip. There was quite a lot of delay to look at the preview. I think this bottleneck needs to be dealt with. Esp. now that OpenShot uses multi-cores to render the final video.
   Also, I noticed that if I render a video with effect (in my case, blur effect - with keyframes), the render happens fast for some time (probably using multi-core) and slows down in the middle (presumably for the effect-application; single-core) and then happens fast again (multi-core again):
   
   

The following screenshot shows my system monitor while the preview-cache was being generated (while it was buffering during playback in OpenShot):

2. I think I saw a discussion somewhere about how OpenShot was currently applying the effects pixel-by-pixel and about how using ImageMagick or GraphicsMagick would probably be faster. Perhaps that needs to be fixed too. But I have to confess, I don't know much about this. [Translation: I don't quite know what I am talking about. 😄]

3. The following scenario (this occurred to me this morning):

• An Image is put on Track 1, and blur effect is applied and it is keyframed to the clip so that it'll appear to blur away (neat artistic choice, right? :D).
• An .svg title is put on Track 2 so as to have a text overlay on top of the blurring clip.
  The following screenshot is a representation of what was going on:
  
  Observations:
  A. The buffering takes quite a lot of time (see 1. and 2.)
  B. If the SVG file is moved after the preview to appear at a different time, the buffer happens again - from scratch. It seems it is regenerating the preview-cache.
  C. Even if we remove the SVG file altogether, and just move the clip where the blur effect is applied, in the timeline, the preview-cache-generation happens all over again. It takes just the same amount of time to buffer, all over again.
  Perhaps, instead of having the sum of all the tracks, as a whole - at a certain point in time - being saved into the preview-cache, it should only save the keyframed clip (where the effect has been applied), in isolation. Maybe some logic should be baked into the preview-cache generation (and, hopefully, retention) that would prevent the need for the preview-cache to be generated all over again, every time a change is made in the time-line.

[N3WWN]

I wonder if there would be any value in creating a couple of projects to test out these theories and use a screen recorder to monitor everything.

Here's what I'm thinking...

Project 1

• 100s long project
• 0-10s one image (transparent PNG or SVG so we can see the underlying video later, when we add it) with no effects, transitions, etc
• 0-20s same image, with slow fades in and out
• 0-30s same image, with no fades - this time apply a static effect to the entire 10s
• 0-40s same image, with no fades - this time keyframe an effect to either apply or remove itself across the entire 10s
• 0-50s same image, with slow fades in and out and the keyframed effect
• 50-100s would be the same as 0-50s, but with a 100s video added to a lower track

Project 2

Same as project one with the following differences:

• Add an audio-only track for the entire 100s
• Instead of an image, use a video clip

Project 3

Same as project two with the following differences:

• Add two more video clips on lower tracks for the entire 100s, perhaps scaling all video tracks to 50% and using gravity to set to upper-left, upper-right, lower-left and lower-right

We would then see how many cores are used for each 10s segment and know which frame is being processed based on the % complete.

I took a look at the code and no code portions jumped out at me that would make applying effects an exclusive computation, but I may need to look harder.

The reason the preview cache needs to be generated again if you change something is that the cache is based on the current frame in the timeline. If you move something on the timeline or adjust a property of something on the timeline, then the cache needs to be updated to reflect that change.

Think of the preview cache as a series of 1 frame wide snapshots through the timeline vertically. You cannot simply take any of those cached frames and move them to another timeline location and expect them to be correct.

There would need to be additional logic to determine if there was a visible (or perhaps even an invisible) change that would require a preview update - and off the top of my head, that may be generating previews of both the old and new location, comparing them and discarding the new frame if it's the same. This would be beneficial if we used disk caching all the time since we could avoid writing the same frame to disk, but we'd still need to either rename the frame's file on disk or we'd need to have a means of mapping the current timeline frame to the disk file.

[peanutbutterandcrackers]

Hmm, yes. I think it would be great. I'll try to do this once I have the time. All the free time I have goes mostly to juggle all the incoming issues, I hardly ever get time to test and or submit the issues I have myself found out. 😫

[robertpartridge]

Hey, I'm sorry to jump into an existing discussion that hasn't seen updates in a year and half, but I'm wondering if there's been any progress happening on this front? I've been evaluating editing software for Linux and I really like openshot (kdenlive crashes constantly and I'm not overly fond of the interface in shotcut). The problem I'm having though with openshot is if I apply any effects to my clips the render speed (4K 60fps) goes from roughly 20fps down to a max of 1.3fps making the render time enormous.

[ferdnyc]

@robertpartridge There hasn't been much progress, no — effects in OpenShot remain a pain point, in terms of speed.

I just submitted #423 to parallelize and streamline Blur (the slowest effect by a wide margin), but even though the new code is more than twice as fast as the current version, that's still really, really slow.

Users on systems with more than my four CPU cores might see additional speedups — I'm not sure, to be honest, whether the new code will parallelize efficiently enough to take advantage of additional cores. I suspect not, actually. I suspect the multithreading may have to be done at a more granular level, in order to really benefit from additional parallelism. But, it's at least a start...

[ferdnyc]

Actually, I've been on kind of a tear, so #424 and #425 optimize Brightness/Contrast and Wave, respectively. (With those effects, I expect having more cores will result in additional speedups, as their algorithms should parallelize extremely well.)

[robertpartridge]

I've got 6 cores / 12 threads so if you need anyone to test I can volunteer as tribute. Just give me a yell on what you need me to do.

[ferdnyc]

@robertpartridge Hmm, you're on linux, so... well, two choices.

1. How do you feel about building your own libopenshot? 😆 (It's... not fun, speaking as someone who does it regularly. After the first time, it's not that bad, but getting all the ducks in a row the first time is a nightmare that's turned more than one person off to the idea of working on it at all, TBH.)

2. Or, I can get an AppImage build together containing the effect changes, for testing.

Either way I may wait until I can tackle a couple more of the effects. Nominations welcome for ones to target first/next.

(I've also submitted an update to Pixelate already (#426). But that was just a code-cleanup thing, since there turned out to be just about zero room for speedups. So I didn't even mention it here.)

[robertpartridge]

As much fun as you make building it myself sound, I think maybe going with an AppImage might be better for now. I don't have tons of time for pulling hair (that is becoming more precious as the years go on) ... I save that for my clients LOL. Whenever you're ready to have me test let me know.

As far as what would I personally like to see working first? Things like Hue or Saturation probably first since you've already worked on Brightness & Contrast.

[ferdnyc]

Next on my list, then!

Meanwhile, I took another pass at the Blur update in #423, and managed to improve the parallelization to the point where now, on my 4-core system, an export that used to take ~ 12 minutes (exporting at 0.58 FPS) now manages a rate of 5.72 FPS, decreasing the export time to 1m 19s. It's literally 10x as fast as it used to be.

And, even better, I now believe it will benefit from additional cores, for even better performance.

[robertpartridge]

If you don't mind me asking - what kind of hardware are you testing with? Right now I'm on a Ryzen 5 3600 but hoping to upgrade to something beefier in the hopefully not to distant future. Wondering to get an idea of relative performance.

[ferdnyc]

In my case, "terrible hardware". My development system is a ca. 2011 HP desktop with a 4-core Intel Core i5-2400 CPU running at 3.10 GHz, and 8GB of (very overtaxed) DDR3 RAM. Coupled with a barest-bones Nvidia GeForce GT 710 that cost $50. It is far from a video-editing powerhouse, but it's the best I can afford, and at least it gives me a good perspective on OpenShot's real-world performance.

[robertpartridge]

I'm glad then that I can at least help with testing. My desktop is currently a Ryzen 5 3600 / 32GB DDR4 / Radeon RX 590. As editing systems go I'd call it "acceptable". I'll be interested to see how well it scales.

[ferdnyc]

Ohhh-kay. This AppImage:

OpenShot-v2.5.0-dev1-optimized_effects-x86_64.AppImage

was built from a project branch with all of the updated effects PRs (423 - 428) merged into it. That's updated versions of:

• Blur
• Brightness/Contrast
• Wave
• Pixelate
• Hue
• Saturation

Pixelate isn't really optimized, it was pretty efficient already and I don't expect much in the way of speedups. The code is cleaner, but that's all.

Saturation and Hue, while already fairly efficient, benefit from the parallelization on my system with roughly 2x performance — and I'd expect at least slightly better results, with more cores.

Wave showed roughly 3x the speed of the old code for me, thanks to some extra inefficiencies I managed to eliminate.

And Blur is the big one, running nearly 10x as fast on my 4-core system. And again, if we're lucky more cores == even greater performance gains, at least slightly. (The improvements surely won't be linear, but it should be something.)

As far as testing, what I've been doing is this:

1. Create a project in OpenShot running with one libopenshot (either my build from the develop branch, or the one I'm testing), set it up with a clip, apply the effect I want to test, and save it to disk.
2. Run an export to gauge the speed/time of the export process.
3. Quit that OpenShot, launch OpenShot running against the other libopenshot, load the saved project file, and perform the same export over again. All I really care about is the final time and FPS stats from the export dialog, though I usually export to different filenames so that I can compare the two output videos to check for any differences with the new code.

Instead of running against different library builds, you'd substitute running either your installed OpenShot, or the AppImage version. But the idea is the same.

Actually, for more representative testing, you might want to use the 2.5.0 release-candidate AppImage as your comparison baseline, rather than your installed version. Just because they'll be built in different ways that could throw off the comparison.

[ferdnyc]

You could of course also experiment with even more involved comparisons, like applying multiple effects and etc... basically anything you can set up and save as a project file, you can load into two different builds of OpenShot and use the Export speed to gauge real-world performance.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.