saveVideo: ffmpeg drops frames (use -framerate on input)

[helmingstay]

The docs for ffmpeg are not great in this respect, but encourage the use of -framerate for input and -r for output rates ("If in doubt use -framerate instead of the input option -r"). If -r is used for both and input -r does not equal output -r, ffmpeg discards large (contiguous) blocks of frames to achieve a correct final ratio. To be honest, ffmpeg's behavior in the case of -r for both input and output is confusing and poorly documented, and should perhaps be avoided altogether.
See, for example, https://video.stackexchange.com/questions/13066/how-to-encode-a-video-at-30-fps-from-images-taken-at-7-fps

My understanding is that there are 2 related issues.

• A: When using still images to make video (as in saveVideo), one can simply specify both input and output to be the same, equal to the desired framerate. This always works in "good" players like vlc.
• B: Some players might be more picky about framerate? If one needs standard 24fps (ffmpeg default?), with 2 plots per second (i.e. output varies slower than input, "slow motion"), ffmpeg will duplicate frames to achieve the desired effect. If one wants 24fps with 48 plots per second ("fast forward"), ffmpeg will drop frames.

Using ffmpeg -framerate 30 -i Rplot%d.png -c:v libx264 -r 30 out.mp4 correctly makes a 30fps video with no dropped frames. For vlc, any combination where -framerate matches -r works fine (and minimizes final size), thus solving A above. I can't comment on whether this solution does not work for other movie players.

[yulijia]

Hi,
I generate mp4 files with these command,

ffmpeg -y -framerate 30 -i Rplot%d.png  -c:v libx264 30f.mp4
ffmpeg -y -framerate 30 -r 30 -i Rplot%d.png -c:v libx264 30fr.mp4
ffmpeg -y -r 30 -i Rplot%d.png -c:v libx264 30r.mp4

You can download the video at there.

Would you tell me how to find the difference between these three videos?
I am not sure if using -framerate is better than using -r.

Thanks.
Lijia

[helmingstay]

Sorry for the slow response. I tried to generate a minimal reproducible example, and I agree, I see no difference (though I did run into some very strange behavior with -r 1).

[helmingstay]

I played with this some more, and I'm dependably get dropped frames using standard animation options.
I've tried to eliminate extra options / constructing a minimal example, and I can't figure out what the root cause is. Still, the results are predictable: -r sometimes drops frames, and -framerate never does.

Here's a paste of the final ffmpeg command (dropped frames):
-- sometimes drops frames
'ffmpeg' -y -r 12 -i Rplot%d.png -hide_banner -c:v libx264 -pix_fmt yuv420p test.mp4
[...]
-- and ffmpeg output indicating drops:
frame= 186 fps= 15 q=-1.0 Lsize= 21648kB time=00:00:15.50 bitrate=11441.2kbits/s dup=0 drop=313
-- never drops
'ffmpeg' -y -framerate 12 -i Rplot%d.png -hide_banner -c:v libx264 -pix_fmt yuv420p test.mp4

I've tried experimenting with other options, like -vsync and -copyts, with no luck. Using -framerate instead of -r reliably works. The ffmpeg docs also seem to agree that -framerate is preferred on the input...

See example here (skip at 0:03-0:04):
https://copy.com/xvSRkByNSuHBXkLI

[yulijia]

Hi, I see your video and find the skip. But I couldn't reproduce the bug.

First, I generate brownian.motion() video by example code in animation package.

saveVideo({
    par(mar = c(3, 3, 1, 0.5), mgp = c(2, 0.5, 0), tcl = -0.3, cex.axis = 0.8, 
        cex.lab = 0.8, cex.main = 1)
    ani.options(interval = 0.05, nmax = 300)
    brownian.motion(pch = 21, cex = 5, col = "red", bg = "yellow")
}, video.name = "BM.mp4", other.opts = "-pix_fmt yuv420p -b 300k")

Then I go to the tmp folder and type the command you provided.

ffmpeg -y -r 12 -i Rplot%d.png -hide_banner -c:v libx264 -pix_fmt yuv420p Rtest.mp4
ffmpeg -y -framerate 12 -i Rplot%d.png -hide_banner -c:v libx264 -pix_fmt yuv420p Ftest.mp4

However, I can not find any difference between the two videos which I generated.

Would you provide some example code using function in animation package to help me reproduce the bug?

Thanks.

[helmingstay]

This took a while to tickle - the drops seem related to color depth, and only trigger in very particular circumstances.

## full example - takes ~10 min & 100MB on new-ish laptop
require(devtools)
install_github('helmingstay/rcpp.cgolr/cgolr')
source('https://raw.githubusercontent.com/helmingstay/rcpp.cgolr/master/mwe.R')

## Some ffmpeg output:
Input stream #0:0 frame changed from size:1280x720 fmt:pal8 to size:1280x720 fmt:rgb24
Input stream #0:0 frame changed from size:1280x720 fmt:rgb24 to size:1280x720 fmt:pal8 drop=157
frame=  566 fps= 38 q=-1.0 Lsize=    7819kB time=00:00:35.25 bitrate=1817.1kbits/s dup=0 drop=434

[yulijia]

Thank you!
I test the demo three times and find there is more than 2 dropout frame when using -r option. To avoid this, we would using -framerate in lieu of -r and test the new option on multiple OS.

Here is my output info.

> source('https://raw.githubusercontent.com/helmingstay/rcpp.cgolr/master/mwe.R')
100%0%
Executing: 'ffmpeg' -y -r 16 -i Rplot%d.png -hide_banner -crf 5 -preset slow -c:v libx264  -pix_fmt yuv420p diamoeba-box.mp4

frame=   70 fps=0.0 q=9.0 size=      38kB time=00:00:00.12 bitrate=2498.0kbits/s    
frame=  133 fps=131 q=9.0 size=     554kB time=00:00:04.06 bitrate=1116.1kbits/s    
frame=  187 fps=123 q=9.0 size=    1457kB time=00:00:07.43 bitrate=1604.4kbits/s    
frame=  232 fps=114 q=9.0 size=    2576kB time=00:00:10.25 bitrate=2058.9kbits/s    
Input stream #0:0 frame changed from size:1280x720 fmt:pal8 to size:1280x720 fmt:rgb24
frame=  258 fps=101 q=9.0 size=    3527kB time=00:00:11.87 bitrate=2432.9kbits/s dup=0 drop=26    
frame=  258 fps= 85 q=9.0 size=    3527kB time=00:00:11.87 bitrate=2432.9kbits/s dup=0 drop=160    
Input stream #0:0 frame changed from size:1280x720 fmt:rgb24 to size:1280x720 fmt:pal8
frame=  258 fps= 73 q=9.0 size=    3527kB time=00:00:11.87 bitrate=2432.9kbits/s dup=0 drop=300    
frame=  258 fps= 64 q=9.0 size=    3527kB time=00:00:11.87 bitrate=2432.9kbits/s dup=0 drop=399    
frame=  282 fps= 62 q=9.0 size=    4611kB time=00:00:13.37 bitrate=2824.3kbits/s dup=0 drop=434
frame=  309 fps= 61 q=9.0 size=    5983kB time=00:00:15.06 bitrate=3253.9kbits/s dup=0 drop=434 
frame=  359 fps= 64 q=9.0 size=    7107kB time=00:00:18.18 bitrate=3201.3kbits/s dup=0 drop=434
frame=  445 fps= 73 q=9.0 size=    7397kB time=00:00:23.56 bitrate=2571.8kbits/s dup=0 drop=434
frame=  543 fps= 82 q=9.0 size=    7585kB time=00:00:29.68 bitrate=2092.9kbits/s dup=0 drop=434
frame=  566 fps= 82 q=-1.0 Lsize=    7769kB time=00:00:35.25 bitrate=1805.6kbits/s dup=0 drop=434    

using -r option

Input stream #0:0 frame changed from size:1280x720 fmt:pal8 to size:1280x720 fmt:rgb24
Input stream #0:0 frame changed from size:1280x720 fmt:rgb24 to size:1280x720 fmt:pal80 drop=104    
frame=  566 fps= 98 q=-1.0 Lsize=    1109kB time=00:00:47.00 bitrate= 193.2kbits/s dup=0 drop=434    

using -framerate option

Input stream #0:0 frame changed from size:1280x720 fmt:pal8 to size:1280x720 fmt:rgb24
Input stream #0:0 frame changed from size:1280x720 fmt:rgb24 to size:1280x720 fmt:pal8

[helmingstay]

Just wanted to check in - is there anything else I can do to help with this?

[yulijia]

Hi,

Sorry for the delay in replying.
Could you help us to test if the development version of animation package can solve your problem?

install.packages('animation', repos = 'http://yihui.name/xran')

Thanks.

[helmingstay]

Yes, thanks! Closing!