Duplicate/skipped frames when re-encoding a GPUImageMovie via GPUImageMovieWriter #1501

Open
giilby opened this Issue Apr 5, 2014 · 3 comments

Comments

Projects
None yet
2 participants
@giilby

giilby commented Apr 5, 2014

Using the approach presented in the readme, I'm blending a GPUImageMovie (sans audio) with a GPUImagePicture and writing the result to a new video file via GPUImageMovieWriter. The output video looks pretty good and is exactly the right length, but there's a small problem: some frames are doubled or skipped, especially at the beginning of the file.

I verified this by creating an input video (H.264, 30 FPS, no audio) that numbers its frames in a repeating 1-10 pattern, e.g. 1, 2, 3, ... 9, 10, 1, 2, etc. When loading this video with GPUImageMovie, playback looks normal, but when I write it out with GPUImageMovieWriter, the first frame is never 1 (it varies from 2 - 5), and there is some visible choppiness every few seconds. Scanning through the files, I've found many sequences where a frame is skipped, followed by two of the same frames, e.g. 4, 5, 7, 7, 8.

Looking at the code, GPUImageMovie's enableSynchronizedEncodingUsingMovieWriter activates a pull-based rendering approach where GPUImageMovieWriter requests each frame from an AVAssetReaderOutput before rendering it and writing it to an AVAssetWriterInput. Shouldn't this approach result in smooth and consistent output, even if other tasks are contending for CPU time (or if the CPU is just plain slow)? I've been digging into the source, and I can't find anything that would cause frames to be lost or duplicated. Granted, I know next to nothing about OpenGLES...could the cause of this issue lurk in the actual rendering of the frames?

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Oct 1, 2014

gilby, this something we've noticed about GPUImageMovieWriter. Have put a considerable amount of hours trying to figure out why there is a seemingly randomness to what frames are skipped or duplicated. I think this has less to do with a 1:1 relationship of what is being played and recorded, but more likely a build up of resources (memory) that requires time to clear which will result in what seems like duplicated frames. I think it is more like playback that is being recorded is stuttering and the recorder is recording that.

My assumption is that a lot of memory handling is going on, I'm sure there is away to optimize this, we just haven't had the time or resources to research this further. Please let me know if you have gotten any leads or ideas on how to fix the issue.

ghost commented Oct 1, 2014

gilby, this something we've noticed about GPUImageMovieWriter. Have put a considerable amount of hours trying to figure out why there is a seemingly randomness to what frames are skipped or duplicated. I think this has less to do with a 1:1 relationship of what is being played and recorded, but more likely a build up of resources (memory) that requires time to clear which will result in what seems like duplicated frames. I think it is more like playback that is being recorded is stuttering and the recorder is recording that.

My assumption is that a lot of memory handling is going on, I'm sure there is away to optimize this, we just haven't had the time or resources to research this further. Please let me know if you have gotten any leads or ideas on how to fix the issue.

@giilby

This comment has been minimized.

Show comment
Hide comment
@giilby

giilby Oct 21, 2014

doringe,

Unfortunately, I was never able to identify the exact cause of this problem with GPUImage. Since my project needed frame-perfect output, I opted for a custom solution using Core Image with AVAssetReader\Writer and AVAssetWriterInputPixelBufferAdaptor.

giilby commented Oct 21, 2014

doringe,

Unfortunately, I was never able to identify the exact cause of this problem with GPUImage. Since my project needed frame-perfect output, I opted for a custom solution using Core Image with AVAssetReader\Writer and AVAssetWriterInputPixelBufferAdaptor.

@tuo tuo referenced this issue Feb 5, 2015

Closed

Fix movie writer #1913

@ofZach ofZach referenced this issue in julapy/ofxiOSVideoWriter Jan 11, 2018

Open

dropping frames #9

joshbernfeld added a commit to joshbernfeld/GPUImage2 that referenced this issue Mar 30, 2018

Resolve dropped CVPixelBuffers in MovieOutput
https://developer.apple.com/documentation/avfoundation/avassetwriterinputpixelbufferadaptor/1388102-append
Documentation: "Do not modify a CVPixelBuffer or its contents after you have passed it to this method."

It must then be a requirement that for each frame we create a new pixel buffer with CVPixelBufferPoolCreatePixelBuffer(). Since we are not doing this, occasionally old pixel buffers are being reused. This is causing dropped frames - in the form of duplicates. These would mostly occur towards the beginning of videos and some would be interspersed throughout the rest of the video. I also noticed this problem impacted certain devices more so than others, usually older devices more than newer devices. With this resolved video encoding is now a 1-1 mapping of frames and 60fps video is supported without issue.

BradLarson/GPUImage#1501
@joshbernfeld

This comment has been minimized.

Show comment
Hide comment
@joshbernfeld

joshbernfeld Mar 30, 2018

@giilby The above commit resolves this issue in GPUImage2

@giilby The above commit resolves this issue in GPUImage2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment