Discarding frames with too much motion blur completely

[lvml]

Hi Georg,

first of all: Thanks a lot for libvidstab, which I am using to my greatest pleasure via ffmpeg! I had tried it long ago with some 1080p material, with mixed results, but recently found that when I use vidstab on 4k videos I recorded, the results are really great, probably because there's so much "spare resolution" in the recordings that a little zooming doesn't hurt the looks of the result as much as with 1080p material.

But now back on topic: I use libvidstab for one additional purpose you probably hadn't in mind when you wrote it: For creating low-noise still images from hand-held video shots under poor lighting conditions. The idea here is that if you don't have a tripod, but want to take a picture in low light, you can substitute for the noise-cancellation that is usually done in-camera for long-exposure still images by first recording some seconds of hand-held 4k-video with relatively high ISO values, then software-stabilize the clip using libvidstab (in "tripod=1" mode), then use a filter like ffmpeg's "hqdn3d" with spatial=0 but temporal=200 or alike parametrization, effectively doing the same "denoising" the camera could have done when using a tripod but from the stabilized hand held video, and save a resulting frame at the end of that clip as a still image.

This already has worked for me remarkebly well, however, there's one catch I'd like to get rid of: If inside the hand-held video there are individual frames with too much motion-blur from unintentional movements, they can significantly lower the quality of the overall result.

Now I wonder whether this could be easily solved by just discarding all frames from the video that have too much motion blur in them.

I understand that libvidstab already does some contrast analysis for the "mincontrast" parameter, and maybe that could be a starting point to implement a minimum threshold for micro-contrast that, if undercut, could cause libvidstab to ignore the whole frame not only for motion estimation, but to drop the frame completely (or replace it by a duplicate of the preceding frame, if that is easier to implement).

What do you think, is such a feature feasible with reasonably low effort?

[magicgoose]

ffmpeg then would have to deal with variable frame rate…
or, for this specific scenario, after skipped frame all subsequent frames can be shifted back in time. I don't know if this is easier, though

[georgmartius]

On Monday 23 March 2015 23:41:30 magicgoose wrote:

ffmpeg then would have to deal with variable frame rate…
I think the only reasonable solution would be to double the frame and move it
with the average camera velocity.

or, for this specific scenario, after skipped frame all subsequent frames
can be shifted back in time. I don't know if this is easier, though
We would get sync problems with audio etc.

---

Reply to this email directly or view it on GitHub:
#24 (comment)

[magicgoose]

We would get sync problems with audio etc.

Of course, but in OP's case this is going to be useful and obviously he doesn't care about time at all — the final result is still picture

[lvml]

Indeed, neither movement nor audio would be of concern, and duplicating a preceding frame when discarding one would be good enough for the purpose.

That's why I think this feature could be realized with reasonably low effort, and open up a great opportunity to do spontaneous night-shots without tripods, even with not-so-big-sensor cameras. :-)

[magicgoose]

duplicating is worse than just discarding the frame for OP, because then it (the doubled frame) will have bigger "weight" in averaging, making noise reduction less efficient

[lvml]

On 03/25/2015 12:34 PM, magicgoose wrote:

duplicating is worse for OP, because then it (the doubled frame) will have bigger "weight" in
averaging, making noise reduction less efficient

Yes, if it was possible to drop a frame completely (essentially making
the output video "shorter") that would be even better.

[georgmartius]

On Wednesday 25 March 2015 04:39:23 Lutz Vieweg wrote:

On 03/25/2015 12:34 PM, magicgoose wrote:

duplicating is worse for OP, because then it (the doubled frame) will have
bigger "weight" in averaging, making noise reduction less efficient

Yes, if it was possible to drop a frame completely (essentially making
the output video "shorter") that would be even better.
I understand. I have to check whether I can do it in ffmpeg easily.
On the other hand, I don't like to spend too much time with a very specific
feature for one type of shoots.

Can you open a feature request on github for it.

Georg

---

Reply to this email directly or view it on GitHub:
#24 (comment)

---- Georg Martius, Tel: +49 177 6413311 -----
------- http://georg.hronopik.de ----------

[lvml]

Can you open a feature request on github for it.

I thought this (#24) was the recommended way to submit a feature request... or what other method is preferred?

[georgmartius]

Sorry, my mistake. This is perfectly fine.

[lvml]

Here is a sample to demonstrate how well the approach already works:

This picture was taken under very difficult lighting conditions, free hand, using ISO 400. It exposes both some image noise (which was reduced, though, when generating the JPEG from the RAW) and at the same time, it was not possible to avoid motion-blur when shooting free-handed (enlarge to watch 1:1 zoom):

The following picture in contrast is a screenshot (at position 9 seconds from start) taken from this original 3840x2160 video, which was taken under the same difficult lighting conditions at ISO 6400, after stabilizing and temporal denoising using this script (enlarge to watch 1:1 zoom):

As you can see, the screen shot from the video is the much better still image.

But the sample video contains a (deliberate) strong movement at 10s into the video, and if you watch the resulting video after stabilization / temporal denoising, you can see how the quality suffers from that. And in many cases you cannot avoid having frequent shakes when shooting free hand. So it would be very good if vidstab could just discard the few frames in the middle of the sample .MP4 file where there is such strong movement / motion blur.