New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

saveVideo: ffmpeg drops frames (use -framerate on input) #74

Closed
helmingstay opened this Issue Jan 5, 2016 · 9 comments

Comments

2 participants
@helmingstay

helmingstay commented Jan 5, 2016

The docs for ffmpeg are not great in this respect, but encourage the use of -framerate for input and -r for output rates ("If in doubt use -framerate instead of the input option -r"). If -r is used for both and input -r does not equal output -r, ffmpeg discards large (contiguous) blocks of frames to achieve a correct final ratio. To be honest, ffmpeg's behavior in the case of -r for both input and output is confusing and poorly documented, and should perhaps be avoided altogether.
See, for example, https://video.stackexchange.com/questions/13066/how-to-encode-a-video-at-30-fps-from-images-taken-at-7-fps

My understanding is that there are 2 related issues.

  • A: When using still images to make video (as in saveVideo), one can simply specify both input and output to be the same, equal to the desired framerate. This always works in "good" players like vlc.
  • B: Some players might be more picky about framerate? If one needs standard 24fps (ffmpeg default?), with 2 plots per second (i.e. output varies slower than input, "slow motion"), ffmpeg will duplicate frames to achieve the desired effect. If one wants 24fps with 48 plots per second ("fast forward"), ffmpeg will drop frames.

Using ffmpeg -framerate 30 -i Rplot%d.png -c:v libx264 -r 30 out.mp4 correctly makes a 30fps video with no dropped frames. For vlc, any combination where -framerate matches -r works fine (and minimizes final size), thus solving A above. I can't comment on whether this solution does not work for other movie players.

@yulijia

This comment has been minimized.

Show comment
Hide comment
@yulijia

yulijia Jan 5, 2016

Collaborator

Hi,
I generate mp4 files with these command,

ffmpeg -y -framerate 30 -i Rplot%d.png  -c:v libx264 30f.mp4
ffmpeg -y -framerate 30 -r 30 -i Rplot%d.png -c:v libx264 30fr.mp4
ffmpeg -y -r 30 -i Rplot%d.png -c:v libx264 30r.mp4

You can download the video at there.

Would you tell me how to find the difference between these three videos?
I am not sure if using -framerate is better than using -r.

Thanks.
Lijia

Collaborator

yulijia commented Jan 5, 2016

Hi,
I generate mp4 files with these command,

ffmpeg -y -framerate 30 -i Rplot%d.png  -c:v libx264 30f.mp4
ffmpeg -y -framerate 30 -r 30 -i Rplot%d.png -c:v libx264 30fr.mp4
ffmpeg -y -r 30 -i Rplot%d.png -c:v libx264 30r.mp4

You can download the video at there.

Would you tell me how to find the difference between these three videos?
I am not sure if using -framerate is better than using -r.

Thanks.
Lijia

@helmingstay

This comment has been minimized.

Show comment
Hide comment
@helmingstay

helmingstay Jan 26, 2016

Sorry for the slow response. I tried to generate a minimal reproducible example, and I agree, I see no difference (though I did run into some very strange behavior with -r 1).

helmingstay commented Jan 26, 2016

Sorry for the slow response. I tried to generate a minimal reproducible example, and I agree, I see no difference (though I did run into some very strange behavior with -r 1).

@helmingstay

This comment has been minimized.

Show comment
Hide comment
@helmingstay

helmingstay Jan 27, 2016

I played with this some more, and I'm dependably get dropped frames using standard animation options.
I've tried to eliminate extra options / constructing a minimal example, and I can't figure out what the root cause is. Still, the results are predictable: -r sometimes drops frames, and -framerate never does.

Here's a paste of the final ffmpeg command (dropped frames):
-- sometimes drops frames
'ffmpeg' -y -r 12 -i Rplot%d.png -hide_banner -c:v libx264 -pix_fmt yuv420p test.mp4
[...]
-- and ffmpeg output indicating drops:
frame= 186 fps= 15 q=-1.0 Lsize= 21648kB time=00:00:15.50 bitrate=11441.2kbits/s dup=0 drop=313
-- never drops
'ffmpeg' -y -framerate 12 -i Rplot%d.png -hide_banner -c:v libx264 -pix_fmt yuv420p test.mp4

I've tried experimenting with other options, like -vsync and -copyts, with no luck. Using -framerate instead of -r reliably works. The ffmpeg docs also seem to agree that -framerate is preferred on the input...

See example here (skip at 0:03-0:04):
https://copy.com/xvSRkByNSuHBXkLI

helmingstay commented Jan 27, 2016

I played with this some more, and I'm dependably get dropped frames using standard animation options.
I've tried to eliminate extra options / constructing a minimal example, and I can't figure out what the root cause is. Still, the results are predictable: -r sometimes drops frames, and -framerate never does.

Here's a paste of the final ffmpeg command (dropped frames):
-- sometimes drops frames
'ffmpeg' -y -r 12 -i Rplot%d.png -hide_banner -c:v libx264 -pix_fmt yuv420p test.mp4
[...]
-- and ffmpeg output indicating drops:
frame= 186 fps= 15 q=-1.0 Lsize= 21648kB time=00:00:15.50 bitrate=11441.2kbits/s dup=0 drop=313
-- never drops
'ffmpeg' -y -framerate 12 -i Rplot%d.png -hide_banner -c:v libx264 -pix_fmt yuv420p test.mp4

I've tried experimenting with other options, like -vsync and -copyts, with no luck. Using -framerate instead of -r reliably works. The ffmpeg docs also seem to agree that -framerate is preferred on the input...

See example here (skip at 0:03-0:04):
https://copy.com/xvSRkByNSuHBXkLI

@helmingstay helmingstay reopened this Jan 27, 2016

@yulijia

This comment has been minimized.

Show comment
Hide comment
@yulijia

yulijia Jan 28, 2016

Collaborator

Hi, I see your video and find the skip. But I couldn't reproduce the bug.

First, I generate brownian.motion() video by example code in animation package.

saveVideo({
    par(mar = c(3, 3, 1, 0.5), mgp = c(2, 0.5, 0), tcl = -0.3, cex.axis = 0.8, 
        cex.lab = 0.8, cex.main = 1)
    ani.options(interval = 0.05, nmax = 300)
    brownian.motion(pch = 21, cex = 5, col = "red", bg = "yellow")
}, video.name = "BM.mp4", other.opts = "-pix_fmt yuv420p -b 300k")

Then I go to the tmp folder and type the command you provided.

ffmpeg -y -r 12 -i Rplot%d.png -hide_banner -c:v libx264 -pix_fmt yuv420p Rtest.mp4
ffmpeg -y -framerate 12 -i Rplot%d.png -hide_banner -c:v libx264 -pix_fmt yuv420p Ftest.mp4

However, I can not find any difference between the two videos which I generated.

Would you provide some example code using function in animation package to help me reproduce the bug?

Thanks.

Collaborator

yulijia commented Jan 28, 2016

Hi, I see your video and find the skip. But I couldn't reproduce the bug.

First, I generate brownian.motion() video by example code in animation package.

saveVideo({
    par(mar = c(3, 3, 1, 0.5), mgp = c(2, 0.5, 0), tcl = -0.3, cex.axis = 0.8, 
        cex.lab = 0.8, cex.main = 1)
    ani.options(interval = 0.05, nmax = 300)
    brownian.motion(pch = 21, cex = 5, col = "red", bg = "yellow")
}, video.name = "BM.mp4", other.opts = "-pix_fmt yuv420p -b 300k")

Then I go to the tmp folder and type the command you provided.

ffmpeg -y -r 12 -i Rplot%d.png -hide_banner -c:v libx264 -pix_fmt yuv420p Rtest.mp4
ffmpeg -y -framerate 12 -i Rplot%d.png -hide_banner -c:v libx264 -pix_fmt yuv420p Ftest.mp4

However, I can not find any difference between the two videos which I generated.

Would you provide some example code using function in animation package to help me reproduce the bug?

Thanks.

@helmingstay

This comment has been minimized.

Show comment
Hide comment
@helmingstay

helmingstay Feb 4, 2016

This took a while to tickle - the drops seem related to color depth, and only trigger in very particular circumstances.

## full example - takes ~10 min & 100MB on new-ish laptop
require(devtools)
install_github('helmingstay/rcpp.cgolr/cgolr')
source('https://raw.githubusercontent.com/helmingstay/rcpp.cgolr/master/mwe.R')
## Some ffmpeg output:
Input stream #0:0 frame changed from size:1280x720 fmt:pal8 to size:1280x720 fmt:rgb24
Input stream #0:0 frame changed from size:1280x720 fmt:rgb24 to size:1280x720 fmt:pal8 drop=157
frame=  566 fps= 38 q=-1.0 Lsize=    7819kB time=00:00:35.25 bitrate=1817.1kbits/s dup=0 drop=434

helmingstay commented Feb 4, 2016

This took a while to tickle - the drops seem related to color depth, and only trigger in very particular circumstances.

## full example - takes ~10 min & 100MB on new-ish laptop
require(devtools)
install_github('helmingstay/rcpp.cgolr/cgolr')
source('https://raw.githubusercontent.com/helmingstay/rcpp.cgolr/master/mwe.R')
## Some ffmpeg output:
Input stream #0:0 frame changed from size:1280x720 fmt:pal8 to size:1280x720 fmt:rgb24
Input stream #0:0 frame changed from size:1280x720 fmt:rgb24 to size:1280x720 fmt:pal8 drop=157
frame=  566 fps= 38 q=-1.0 Lsize=    7819kB time=00:00:35.25 bitrate=1817.1kbits/s dup=0 drop=434
@yulijia

This comment has been minimized.

Show comment
Hide comment
@yulijia

yulijia Feb 5, 2016

Collaborator

Thank you!
I test the demo three times and find there is more than 2 dropout frame when using -r option. To avoid this, we would using -framerate in lieu of -r and test the new option on multiple OS.

Here is my output info.

> source('https://raw.githubusercontent.com/helmingstay/rcpp.cgolr/master/mwe.R')
100%0%
Executing: 'ffmpeg' -y -r 16 -i Rplot%d.png -hide_banner -crf 5 -preset slow -c:v libx264  -pix_fmt yuv420p diamoeba-box.mp4

frame=   70 fps=0.0 q=9.0 size=      38kB time=00:00:00.12 bitrate=2498.0kbits/s    
frame=  133 fps=131 q=9.0 size=     554kB time=00:00:04.06 bitrate=1116.1kbits/s    
frame=  187 fps=123 q=9.0 size=    1457kB time=00:00:07.43 bitrate=1604.4kbits/s    
frame=  232 fps=114 q=9.0 size=    2576kB time=00:00:10.25 bitrate=2058.9kbits/s    
Input stream #0:0 frame changed from size:1280x720 fmt:pal8 to size:1280x720 fmt:rgb24
frame=  258 fps=101 q=9.0 size=    3527kB time=00:00:11.87 bitrate=2432.9kbits/s dup=0 drop=26    
frame=  258 fps= 85 q=9.0 size=    3527kB time=00:00:11.87 bitrate=2432.9kbits/s dup=0 drop=160    
Input stream #0:0 frame changed from size:1280x720 fmt:rgb24 to size:1280x720 fmt:pal8
frame=  258 fps= 73 q=9.0 size=    3527kB time=00:00:11.87 bitrate=2432.9kbits/s dup=0 drop=300    
frame=  258 fps= 64 q=9.0 size=    3527kB time=00:00:11.87 bitrate=2432.9kbits/s dup=0 drop=399    
frame=  282 fps= 62 q=9.0 size=    4611kB time=00:00:13.37 bitrate=2824.3kbits/s dup=0 drop=434
frame=  309 fps= 61 q=9.0 size=    5983kB time=00:00:15.06 bitrate=3253.9kbits/s dup=0 drop=434 
frame=  359 fps= 64 q=9.0 size=    7107kB time=00:00:18.18 bitrate=3201.3kbits/s dup=0 drop=434
frame=  445 fps= 73 q=9.0 size=    7397kB time=00:00:23.56 bitrate=2571.8kbits/s dup=0 drop=434
frame=  543 fps= 82 q=9.0 size=    7585kB time=00:00:29.68 bitrate=2092.9kbits/s dup=0 drop=434
frame=  566 fps= 82 q=-1.0 Lsize=    7769kB time=00:00:35.25 bitrate=1805.6kbits/s dup=0 drop=434    

using -r option

Input stream #0:0 frame changed from size:1280x720 fmt:pal8 to size:1280x720 fmt:rgb24
Input stream #0:0 frame changed from size:1280x720 fmt:rgb24 to size:1280x720 fmt:pal80 drop=104    
frame=  566 fps= 98 q=-1.0 Lsize=    1109kB time=00:00:47.00 bitrate= 193.2kbits/s dup=0 drop=434    

using -framerate option

Input stream #0:0 frame changed from size:1280x720 fmt:pal8 to size:1280x720 fmt:rgb24
Input stream #0:0 frame changed from size:1280x720 fmt:rgb24 to size:1280x720 fmt:pal8
Collaborator

yulijia commented Feb 5, 2016

Thank you!
I test the demo three times and find there is more than 2 dropout frame when using -r option. To avoid this, we would using -framerate in lieu of -r and test the new option on multiple OS.

Here is my output info.

> source('https://raw.githubusercontent.com/helmingstay/rcpp.cgolr/master/mwe.R')
100%0%
Executing: 'ffmpeg' -y -r 16 -i Rplot%d.png -hide_banner -crf 5 -preset slow -c:v libx264  -pix_fmt yuv420p diamoeba-box.mp4

frame=   70 fps=0.0 q=9.0 size=      38kB time=00:00:00.12 bitrate=2498.0kbits/s    
frame=  133 fps=131 q=9.0 size=     554kB time=00:00:04.06 bitrate=1116.1kbits/s    
frame=  187 fps=123 q=9.0 size=    1457kB time=00:00:07.43 bitrate=1604.4kbits/s    
frame=  232 fps=114 q=9.0 size=    2576kB time=00:00:10.25 bitrate=2058.9kbits/s    
Input stream #0:0 frame changed from size:1280x720 fmt:pal8 to size:1280x720 fmt:rgb24
frame=  258 fps=101 q=9.0 size=    3527kB time=00:00:11.87 bitrate=2432.9kbits/s dup=0 drop=26    
frame=  258 fps= 85 q=9.0 size=    3527kB time=00:00:11.87 bitrate=2432.9kbits/s dup=0 drop=160    
Input stream #0:0 frame changed from size:1280x720 fmt:rgb24 to size:1280x720 fmt:pal8
frame=  258 fps= 73 q=9.0 size=    3527kB time=00:00:11.87 bitrate=2432.9kbits/s dup=0 drop=300    
frame=  258 fps= 64 q=9.0 size=    3527kB time=00:00:11.87 bitrate=2432.9kbits/s dup=0 drop=399    
frame=  282 fps= 62 q=9.0 size=    4611kB time=00:00:13.37 bitrate=2824.3kbits/s dup=0 drop=434
frame=  309 fps= 61 q=9.0 size=    5983kB time=00:00:15.06 bitrate=3253.9kbits/s dup=0 drop=434 
frame=  359 fps= 64 q=9.0 size=    7107kB time=00:00:18.18 bitrate=3201.3kbits/s dup=0 drop=434
frame=  445 fps= 73 q=9.0 size=    7397kB time=00:00:23.56 bitrate=2571.8kbits/s dup=0 drop=434
frame=  543 fps= 82 q=9.0 size=    7585kB time=00:00:29.68 bitrate=2092.9kbits/s dup=0 drop=434
frame=  566 fps= 82 q=-1.0 Lsize=    7769kB time=00:00:35.25 bitrate=1805.6kbits/s dup=0 drop=434    

using -r option

Input stream #0:0 frame changed from size:1280x720 fmt:pal8 to size:1280x720 fmt:rgb24
Input stream #0:0 frame changed from size:1280x720 fmt:rgb24 to size:1280x720 fmt:pal80 drop=104    
frame=  566 fps= 98 q=-1.0 Lsize=    1109kB time=00:00:47.00 bitrate= 193.2kbits/s dup=0 drop=434    

using -framerate option

Input stream #0:0 frame changed from size:1280x720 fmt:pal8 to size:1280x720 fmt:rgb24
Input stream #0:0 frame changed from size:1280x720 fmt:rgb24 to size:1280x720 fmt:pal8
@helmingstay

This comment has been minimized.

Show comment
Hide comment
@helmingstay

helmingstay Feb 9, 2016

Just wanted to check in - is there anything else I can do to help with this?

helmingstay commented Feb 9, 2016

Just wanted to check in - is there anything else I can do to help with this?

yulijia added a commit that referenced this issue Feb 17, 2016

Merge pull request #78 from yulijia/master
Fix ( issue #74 ): ffmpeg drops frames
Build and check package: all examples with CPI or elapsed time < 5s
@yulijia

This comment has been minimized.

Show comment
Hide comment
@yulijia

yulijia Feb 17, 2016

Collaborator

Hi,

Sorry for the delay in replying.
Could you help us to test if the development version of animation package can solve your problem?

install.packages('animation', repos = 'http://yihui.name/xran')

Thanks.

Collaborator

yulijia commented Feb 17, 2016

Hi,

Sorry for the delay in replying.
Could you help us to test if the development version of animation package can solve your problem?

install.packages('animation', repos = 'http://yihui.name/xran')

Thanks.

@helmingstay

This comment has been minimized.

Show comment
Hide comment
@helmingstay

helmingstay Feb 22, 2016

Yes, thanks! Closing!

helmingstay commented Feb 22, 2016

Yes, thanks! Closing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment