Video overlay on top of live feed

[cosmicsalad]

Hi!
So what I'd like to is overlay a short looping video as a texture on top of the live feed previewImage (cgImage) from the CameraFilterView.swift example on MetalPetal, then export the recording later. I've noticed in other threads on here the mention of using PlayerVideoOutput to do so.

Is there a possible simple usage example on how this would work when the live feed is a cgImage rather than an AVPlayer?

Would this be a good way to do it? -

• Setup the overlay video as a separate MTIImage on top of the previewImage feed so it constantly plays
• Then after recording, add the overlay video as a track into MTIVideoComposition when showing the player?

I think I'm confused on the implementation methods here though. Just want to live record with a looping MP4 playing on top.

[YuAo]

• Use an AVPlayer to play the overlay video.
• Attach a PlayerVideoOutput to the player. The PlayerVideoOutput outputs the video frames.
• Use a MTIMultilayerCompositingFilter(ObjC)/MultilayerCompositingFilter(swift) to overlay the video frame (CVPixelBuffer -> MTIImage -> MultilayerCompositingFilter.layers) with the live feed (CGImage -> MTIImage -> MultilayerCompositingFilter.backgroundImage).
• Render the output image using a MTIContext.

MTIContext, MTIImage, MTIMultilayerCompositingFilter are parts of the MetalPetal framework. https://github.com/MetalPetal/MetalPetal

[isamercan]

@YuAo is there any sample on MetalPetal
When I check the sample project ı could not see it.

[YuAo]

@isamercan No.

• There are examples about MTIMultilayerCompositingFilter/MultilayerCompositingFilter and Camera.
• The interface of PlayerVideoOutput is quite simple.

Is there anything that confuses you?

[isamercan]

@YuAo Thanks. I have made many variations by using examples and solutions from issues. But Could not make succeed on put two video sources (one of them should have MTIBlendFilter)

[YuAo]

@isamercan Oh, that is in fact simpler. You can follow the video processor example and refer to this thread: MetalPetal/MetalPetal#302

The pseudo code should be like:

let composition = AVMutableComposition(...)
for video in yourVideos {
    let videoTrack = composition.addMutableTrack(...)
    let videoAsset = AVURLAsset(url: video.url, ...)
    let originalVideoTrack = videoAsset.tracks(withMediaType: .video)[0] //load the original track
    videoTrack.insertTimeRange(of: originalVideoTrack ...)
    withExtendedLifetime(videoAsset, ...)
}
...
let videoComposition = MTIVideoComposition(asset: composition, context: renderContext, queue: DispatchQueue.main, filter: { request in
    let images = request.sourceImages....
    let filter = MultilayerCompositingFilter()
    filter.layers = ... //create layers from images, apply blend modes
    filter.backgroundImage = ....
    return filter.outputImage!
})
// use videoComposition for play / export
...

[isamercan]

@YuAo you are the best. Thank you very much 🙏

[isamercan]

@YuAo How can I apply blend mode for added second source?

[YuAo]

How can I apply blend mode for added second source?

You can use the blendMode(_:) method of MultilayerCompositingFilter.Layer:

https://github.com/MetalPetal/MetalPetal/blob/58b226325e52ee13d18e869f28912522dc476ca1/Frameworks/MetalPetal/Filters/MultilayerCompositingFilter.swift#L163

[isamercan]

@YuAo Thanks I added like that.
MultilayerCompositingFilter.Layer(content: DemoImages.snow_effect) .frame(CGRect(x: 0, y: 0, width: 1, height: 1), layoutUnit: .fractionOfBackgroundSize) .blendMode(.overlay)

[isamercan]

@YuAo When I use just one asset inside MTIVideoComposition block I can access request.anySourceImage! and get successfull result. But I can not access multiple sources like request.sourceImages[composition.tracks.last?.trackID ?? 0] .
What do you suggest to me?

func updateVideoOverlay(_ url: URL) {
            let composition = AVMutableComposition()
            let videoOverlayPath = Bundle.main.path(forResource: "myMovie", ofType: "mp4")
            let videoOverlayPathUrl = NSURL.fileURL(withPath: videoOverlayPath!)
            
            let asset = AVURLAsset(url: url, options: [AVURLAssetPreferPreciseDurationAndTimingKey: true])
            let overlayAsset = AVURLAsset(url: videoOverlayPathUrl, options: [AVURLAssetPreferPreciseDurationAndTimingKey: true])
            let presentationSize = CGSize(width: 1080, height: 1920)
            
            let videoTrackA = composition.addMutableTrack(withMediaType: .video, preferredTrackID: composition.unusedTrackID())
            let videoTrackB = composition.addMutableTrack(withMediaType: .video, preferredTrackID: composition.unusedTrackID())

            do {
                try videoTrackA!.insertTimeRange(asset.tracks(withMediaType: .video).first!.timeRange, of: asset.tracks(withMediaType: .video).first!, at: .zero)
                try videoTrackB!.insertTimeRange(overlayAsset.tracks(withMediaType: .video).first!.timeRange, of: overlayAsset.tracks(withMediaType: .video).first!, at: .zero)
            } catch let error {
                print(error)
            }
            
            guard composition.tracks.count >= 2 else {
                return
            }
            
            let videoComposition = MTIVideoComposition(asset: composition, context: renderContext, queue: DispatchQueue.main, filter: { request in
                
                
                guard let bgImage: MTIImage = request.sourceImages[composition.tracks.first?.trackID ?? 0] else {
                    return MTIImage.black
                }
                guard let overlayImage: MTIImage = request.sourceImages[composition.tracks.last?.trackID ?? 0] else {
                    return MTIImage.black
                }

                let filter = MultilayerCompositingFilter()
                filter.layers = [ MultilayerCompositingFilter.Layer(content: overlayImage).frame(CGRect(x: 0, y: 0, width: 1, height: 1), layoutUnit: .fractionOfBackgroundSize)
                                    .blendMode(.overlay)]
                filter.inputBackgroundImage = bgImage
                return filter.outputImage!
            
            })
            let playerItem = AVPlayerItem(asset: asset)
            videoComposition.renderSize = presentationSize
            
            playerItem.videoComposition = videoComposition.makeAVVideoComposition()
            self.videoComposition = videoComposition
            videoAsset = asset
            videoPlayer = AVPlayer(playerItem: playerItem)
            videoPlayer?.play()
        }

[YuAo]

@isamercan The player is not playing the two-track video.

[isamercan]

@YuAo I dont want to play 2 tracks. I want to apply second video as blend mode over mode. I could not add request.sourceimages[2] to filter.backgroundimage

[YuAo]

@isamercan I don't understand. If you are not going to give the player a two-track asset, how can it blend one track over another? The MTIVideoComposition is driven by the AVPlayer through AVVideoComposition.

[isamercan]

@YuAo I can. apply overlay image to video by using layer blend mode or MTIBlendFilter but when i want to apply overlay video asset to original video asset then I am not successful.

let filter = MultilayerCompositingFilter()
                filter.layers = [ MultilayerCompositingFilter.Layer(content: **request.sourceImages[overlay asset trackID]**).frame(CGRect(x: 0, y: 0, width: 1, height: 1), layoutUnit: .fractionOfBackgroundSize)
                                    .blendMode(.overlay)]
                filter.inputBackgroundImage = **request.sourceImages[2]** there should be original video asset
                
                return filter.outputImage!

It does not return images
request.sourceImages[overlay asset trackID]
request.sourceImages[2]

[YuAo]

@isamercan Because your player plays the one-track asset. It cannot get anything about the overlay track so there's only one item in the sourceImages.

The MTIVideoComposition is driven by the AVPlayer through AVVideoComposition.

You must understand that the request.sourceImages is provided by the AVPlayer.

---

In your case, the player should play composition instead of asset.

[isamercan]

@YuAo Thanks. for your great. help 🙏🏻. Thanks for your patience. It's done.🎉 MetalPetal and VideoIO are really great APIs.

[isamercan]

@YuAo is there any option to loop any one of the sources
for example assetOne time range: 5 seconds, assetTwo's time range 8 seconds and composition final time range should be 15 seconds. how can I loop assets in the filter block

[YuAo]

@isamercan Yes, you can compose that AVMutableComposition any way you want.

[isamercan]

@YuAo but short duration source stops. how to start and loop until end of final time range?

[YuAo]

@isamercan I think you should read the "Editing" part of the AVFoundation Programming Guide. https://developer.apple.com/library/archive/documentation/AudioVideo/Conceptual/AVFoundationPG/Articles/03_Editing.html#//apple_ref/doc/uid/TP40010188-CH8-SW1

You can also google for "AVMutableComposition loop"

[isamercan]

@YuAo Can you give a hint?

[isamercan]

@YuAo

try videoTrack?.insertTimeRange(
                        CMTimeRangeMake(start: .zero, duration: originalVideoTrack.asset!.duration),
                        of: originalVideoTrack,
                        at: originalVideoTrack.asset!.duration)

I could able to add it as in the code line seen above. but Im now sure its the best way to set a end duration and looping?

[YuAo]

@isamercan I think you can refer to this answer: https://stackoverflow.com/questions/20211694/loop-avmutablecompositiontrack

[isamercan]

@YuAo

let videoComposition = MTIVideoComposition(asset: composition, context: renderContext, queue: DispatchQueue.main, filter: { request in
      
      
      guard let sourceImage = request.anySourceImage else {
        return MTIImage.black
      }
      
      guard let overlayImage = request.sourceImages[composition.tracks.last!.trackID] else {
        return MTIImage.black
      }
      
      let filter = MultilayerCompositingFilter()
      filter.layers = [ MultilayerCompositingFilter.Layer(content: overlayImage).frame(CGRect(x: 0, y: 0, width: 1, height: 1), layoutUnit: .fractionOfBackgroundSize)
                          .blendMode(.screen)]
     

filter.inputBackgroundImage = request.sourceImages[composition.tracks.first!.trackID]
      return filter.outputImage!      
    })

when I checked your example project that I found this block

FilterParameter(name: "Color Lookup Intensity", defaultValue: 1, sliderRange: 0...1, updater: { filter, intensity in
                filter.layers[5] = filter.layers[5].opacity(intensity)

I want to change the overlay video source while playing the BG video.
Is there any way to change the overlay without blinking eye.

[YuAo]

@isamercan If you have a few overlay videos, you can add them all to the composition and use trackID to switch between them in the filter block. Most iOS device can handle compositions with 16 video tracks.

If there are a lot of possible overlay videos or the video can be chosen by a user, you'll have to decode and synchronize the video playback yourself. This is beyond what MTIVideoComposition can do.

[isamercan]

@YuAo Yes that's right overlays can be chosen by users while playing composition. Could I change the overlay video by using this method?

FilterParameter(name: "Color Lookup Intensity", defaultValue: 1, sliderRange: 0...1, updater: { filter, intensity in
                filter.layers[5] = filter.layers[5].opacity(intensity)

[YuAo]

@isamercan No.

[isamercan]

@YuAo so the Composition video will start beginning every time?

[YuAo]

@isamercan

• If you can use the AVAssetReader, VTDecompressionSession or something like ffmpeg to decode the overlay video frames and synchronize them with the video playback, you can gaplessly change the overlay video.

• If the count of your overlay videos is less than 16, you can add them all to the composition and switch between them based on their trackID.

• If you choose to recreate the composition each time you change the overlay. You don't have to start from beginning every time you change the composition - you can seek the video.

[isamercan]

@YuAo Thanks. I preferred the last one.