New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blend two movie inputs, maintaining sync #261

Closed
LeeMatthewHiggins opened this Issue Jul 13, 2012 · 10 comments

Comments

Projects
None yet
5 participants
@LeeMatthewHiggins

LeeMatthewHiggins commented Jul 13, 2012

It would be good to be able to have videos as inputs to the two input filters....
I'm trying to do something like...

GPUImageMovie *colourMovie; //init correctly etc...
GPUImageMovie *reflectionMovie;

GPUImageScreenBlendFilter* reflectionFilter;

[colourMovie addTarget:reflectionFilter];
[reflectionMovie addTarget:reflectionFilter];

[reflectionFilter addTarget:self.viewer];

[colourMovie startProcessing];
[reflectionMovie startProcessing];

This sometimes works if I change the order in which the movies are started, but has threading sync issues as both movies are driving the processing.

I get an assert: Incomplete filter FBO: most of the time.

Is there any way to do this? Do I need to add some semaphores to the rendering?

@BradLarson

This comment has been minimized.

Show comment
Hide comment
@BradLarson

BradLarson Jul 13, 2012

Owner

There currently is a bug in the blend filters where they won't initialize their size properly if they get fed a frame from the second input before they get one from the first input. To work around this, you can use -forceProcessingAtSize: on the blend filter and manually specify a movie size before either of the movies start processing.

However, this won't address a more fundamental issue of different playback frame rates for the two movies, which will quickly cause them to fall out of sync. I haven't spent much time working on this, and I'd welcome any contributions you'd have to help fix this timing issue.

Owner

BradLarson commented Jul 13, 2012

There currently is a bug in the blend filters where they won't initialize their size properly if they get fed a frame from the second input before they get one from the first input. To work around this, you can use -forceProcessingAtSize: on the blend filter and manually specify a movie size before either of the movies start processing.

However, this won't address a more fundamental issue of different playback frame rates for the two movies, which will quickly cause them to fall out of sync. I haven't spent much time working on this, and I'd welcome any contributions you'd have to help fix this timing issue.

@amitm02

This comment has been minimized.

Show comment
Hide comment
@amitm02

amitm02 Jul 13, 2012

hi brad,
my solution for this was

  1. create an AVComposition containing two tracks, one for each video.
  2. create a custom asset reader that read from the track with the earlier frame. (so that will contain 2 frames of the same timestamp from each track).
  3. create a custom blender which only process the frames if the two frames are with the same time stamp.

amitm02 commented Jul 13, 2012

hi brad,
my solution for this was

  1. create an AVComposition containing two tracks, one for each video.
  2. create a custom asset reader that read from the track with the earlier frame. (so that will contain 2 frames of the same timestamp from each track).
  3. create a custom blender which only process the frames if the two frames are with the same time stamp.
@BradLarson

This comment has been minimized.

Show comment
Hide comment
@BradLarson

BradLarson Jul 17, 2012

Owner

I updated the blend filters over the weekend to only process once a frame has been fed to them from both inputs. Do you still have this problem with the latest code from the repository?

Owner

BradLarson commented Jul 17, 2012

I updated the blend filters over the weekend to only process once a frame has been fed to them from both inputs. Do you still have this problem with the latest code from the repository?

@LeeMatthewHiggins

This comment has been minimized.

Show comment
Hide comment
@LeeMatthewHiggins

LeeMatthewHiggins Jul 17, 2012

Looks like you added the same fix I tried (except I broke still images) Much better. I have 6 video inputs to merge, so I've decided to pre-process and combine some of my videos into a tiled large video and use crop filters to extract the tiles, I still have to combine some different movie files and this is now working well on a latest gen iPod. cheers.

LeeMatthewHiggins commented Jul 17, 2012

Looks like you added the same fix I tried (except I broke still images) Much better. I have 6 video inputs to merge, so I've decided to pre-process and combine some of my videos into a tiled large video and use crop filters to extract the tiles, I still have to combine some different movie files and this is now working well on a latest gen iPod. cheers.

@LeeMatthewHiggins

This comment has been minimized.

Show comment
Hide comment
@LeeMatthewHiggins

LeeMatthewHiggins Jul 18, 2012

Unfortunately there is still a big problem, everything works fine if the device can keep up, but when I run it on my lower end devices the videos get out of sync. So if one video takes longer to decode than another the quicker video will race ahead. we need some way to block the asset readers until both frames have been received by the two input filter. I tried adding adding semaphores to sync up the two inputs but as they are both on the same queue they will create a deadlock. any ideas?

LeeMatthewHiggins commented Jul 18, 2012

Unfortunately there is still a big problem, everything works fine if the device can keep up, but when I run it on my lower end devices the videos get out of sync. So if one video takes longer to decode than another the quicker video will race ahead. we need some way to block the asset readers until both frames have been received by the two input filter. I tried adding adding semaphores to sync up the two inputs but as they are both on the same queue they will create a deadlock. any ideas?

@BradLarson

This comment has been minimized.

Show comment
Hide comment
@BradLarson

BradLarson Jul 18, 2012

Owner

Keeping multiple GPUImageMovie instances in sync is going to require some sort of timing mechanism, and I don't have anything like that in place (which is why you can't play a movie at its original speed). You're welcome to work on this, but I'm not doing anything with movies right now, so I won't get to this for a while.

Owner

BradLarson commented Jul 18, 2012

Keeping multiple GPUImageMovie instances in sync is going to require some sort of timing mechanism, and I don't have anything like that in place (which is why you can't play a movie at its original speed). You're welcome to work on this, but I'm not doing anything with movies right now, so I won't get to this for a while.

@BradLarson

This comment has been minimized.

Show comment
Hide comment
@BradLarson

BradLarson Sep 5, 2012

Owner

I just added a means of telling input movies to respect the original playback rate, which should address this. See the new playAtActualSpeed property on GPUImageMovie.

Owner

BradLarson commented Sep 5, 2012

I just added a means of telling input movies to respect the original playback rate, which should address this. See the new playAtActualSpeed property on GPUImageMovie.

@EfeAcikgoz

This comment has been minimized.

Show comment
Hide comment
@EfeAcikgoz

EfeAcikgoz Feb 18, 2014

Is there any other way to handle timing in GPUImage right now other than playAtActualSpeed, if not can you direct us in the right direction to implement it?

EfeAcikgoz commented Feb 18, 2014

Is there any other way to handle timing in GPUImage right now other than playAtActualSpeed, if not can you direct us in the right direction to implement it?

@BradLarson

This comment has been minimized.

Show comment
Hide comment
@BradLarson

BradLarson Feb 18, 2014

Owner

@efesus This would need to be handled by the two-input filters, where it would need to properly order frames based on their timestamps, but nothing of the sort is currently implemented.

Owner

BradLarson commented Feb 18, 2014

@efesus This would need to be handled by the two-input filters, where it would need to properly order frames based on their timestamps, but nothing of the sort is currently implemented.

@tuo

This comment has been minimized.

Show comment
Hide comment
@tuo

tuo Jan 20, 2015

hi @efesus, I think this one might help you on this: GPUImage Video Merging: Timestamp Synchronization

tuo commented Jan 20, 2015

hi @efesus, I think this one might help you on this: GPUImage Video Merging: Timestamp Synchronization

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment