Color of exported video differs from the input video

[wilfredbtan]

First of all thank you for this awesome library, it has helped me a lot. My issue is that the colors of the exported video are slightly different from the input video although I did not specify any custom colorspaces. I modified the demo project to reproduce the issue as shown below.

Checklist

• I've read the README
• If possible, I've reproduced the issue using the master branch of this repo
• I've searched for existing GitHub issues

Environment

Info	Value
MetalPetal Version	1.22.0
Integration Method	CocoaPods
Platform & Version	iOS 14.5
Device	iPhone X

Steps to Reproduce

To see the color change in export, you can replace the function of VideoProcessorView#updateVideoUrl(url: URL) in the VideoProcessorView file with the following code and export the video.

• This is a stripped down version of the demo which exports a brown (hex string #B5463D) square on a blue background.
• In the Video Processing screen, you can select the brown square test video I attached below and export it.

        func updateVideoURL(_ url: URL) {
            let asset = AVURLAsset(url: url, options: [AVURLAssetPreferPreciseDurationAndTimingKey: true])
            let presentationSize = CGSize(width: 1080, height: 1920)
            let outputFilter = MultilayerCompositingFilter()

            let videoComposition = MTIVideoComposition(asset: asset, context: renderContext, queue: DispatchQueue.main, filter: { request in
                guard let sourceImage = request.anySourceImage else {
                    return MTIImage.white
                }

                // Add a 7 sec brown square video.
                let videoWidth: CGFloat = 500
                let videoHeight: CGFloat = 500
                let videoRect = CGRect(x: presentationSize.width / 2 - videoWidth / 2,
                                       y: presentationSize.height / 2 - videoHeight / 2,
                                       width: videoWidth,
                                       height: videoHeight)
                let videoLayer = MultilayerCompositingFilter.Layer(content: sourceImage).frame(videoRect, layoutUnit: .pixel)
                outputFilter.layers = [ videoLayer ]
                
                // Make the background blue
                let color = MTIColor(red: 0, green: 0, blue: 1, alpha: 1)
                let backgroundImage = MTIImage(color: color, sRGB: true, size: presentationSize)
                outputFilter.inputBackgroundImage = backgroundImage
                return outputFilter.outputImage!
            })
            
            videoComposition.renderSize = presentationSize
            
            let playerItem = AVPlayerItem(asset: asset)
            playerItem.videoComposition = videoComposition.makeAVVideoComposition()
            self.videoComposition = videoComposition
            videoAsset = asset
            videoPlayer = AVPlayer(playerItem: playerItem)
            videoPlayer?.play()
        }

Expected behavior

I expected the color of the exported video to be the same as the input video.

Actual behavior

Firstly, the color of the brown square in the preview video is very slightly different (#B5463F compared to the input #B5463D) but it is unnoticeable and thus not an issue. However, the exported video's brown square is quite different (#AB3B40) and is noticeable when the input video and exported videos are placed side by side.

Input brown square 7s video
https://user-images.githubusercontent.com/21056777/132044499-1201b67b-138d-49bd-ba1c-a3b7505880a4.mov

Exported brown square on blue background video
https://user-images.githubusercontent.com/21056777/132044571-568fe3ec-46d1-4b64-8ef6-a87f9dd96525.mp4

Screenshot of preview video

I intend to use MetalPetal in my video collage app which allows users to add stickers (images), text, etc onto a video and export it. Thus, it is important that the video colors do not deviate too much from the original. Even though this example uses a single video on a background, the same issue is observed when I try to add images onto the video using the MultilayerCompositingFilter.Layer method similar to the demo.

What I've tried so far

I suspected that this was a colorspace issue and thus tried to use a custom colorspace when initializing the images like this:

// Tried different colorspaces like sRGB, linearSRGB etc
let options = MTICGImageLoadingOptions(colorSpace: CGColorSpace(name: CGColorSpace.displayP3))
let colorMtiImage = MTIImage(cgImage: cgImage, options: options)

I also tried applying a colorspace conversion filter like so:

let curveFilter = MTIRGBColorSpaceConversionFilter()
curveFilter.outputPixelFormat = .bgra8Unorm_srgb
return curveFilter.outputImage

But the output colors are still different from the input video.

If I'm not wrong, iOS' default colorspace is sRGB and this should be used by default when a MTIImage is initialized using a CGImage according to the docs, so I'm not sure what is causing the colors to be slightly off. Thanks once again for maintaining this library and I would appreciate it if you could offer some advice on how to fix this issue.

[YuAo]

1. let backgroundImage = MTIImage(color: color, sRGB: true, size: presentationSize) the sRGB parameter here is not wether the color is in RGB color space, its wether MetalPetal should use a sRGB texture format. When a sampler sample a pixel from a sRGB texture, it coverts the sRGB values to linearSRGB values.

   I don't think that you actually want a sRGB texture here. However since your color is blue, this does not actually making any differences (blue has the same color value in sRGB and linearSRGB color space).

2. Color spaces and transfer functions:

   I'm not sure what makes the preview and export different. They should have the similar results. This may caused by some unmatched default parameters between the AVFoundation render system and the export system.

   Also, the input and output color space in MTIVideoComposition is not handled. The color space for the video frame can be accessed using CVImageBufferCreateColorSpaceFromAttachments(CVBufferGetAttachments(pixelBuffer, .shouldPropagate)!). However this information has not been taken into account in MTIVideoComposition for now. This should be a future improvement.

   I'll take a deeper look when time permits.

   However heres how to make export work for your case:

    let videoComposition = videoComposition.makeAVVideoComposition().mutableCopy() as! AVMutableVideoComposition
    videoComposition.colorPrimaries = AVVideoColorPrimaries_ITU_R_709_2
    videoComposition.colorYCbCrMatrix = AVVideoYCbCrMatrix_ITU_R_709_2
    videoComposition.colorTransferFunction = AVVideoTransferFunction_ITU_R_709_2
    configuration.videoComposition = videoComposition
   

[wilfredbtan]

@YuAo Thanks for the quick response. I misunderstood the usage for the sRGB parameter. The fix for 2. works well in both the demo and my app. 😬