Skip to content
GPU Video Effects With tvOS in Realtime
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
GPUVideoEffectsWithTVOS.xcodeproj
GPUVideoEffectsWithTVOS
.gitignore
AVSampleBufferDisplayLayer.h
README.md

README.md

GPU Video Effects With TVOS

TL;DR

Shows you how to run realtime effects over a streaming video file on the GPU on Apple tvOS.

The What

This tutorial will show you how to run realtime effects on a video that is being streamed from the internet at 60 frames per second.

We are not going to use any 3rd party libraries for this. This is all native.

The Why

On a project I was asked to blur a video that was playing on the screen (1920 x 1080). I first thought ok, I’ll just use a UIVisualEffect view, this did the trick, but the designer wasn’t happy with it, he wanted more control. So I went and found a load of ways that didn’t work, and one that works amazingly well.

Demos

There are 5 demos in this projects

  • Realtime Blur with MPS (Metal Performance Shaders)
  • Realtime Blur with A Custom Gaussian Function (This is slow)
  • Realtime Image Blending of a video frame and an image
  • Realtime color blindness simulation
    • This is really great if you have a UI over the video and want to see how it looks to people with different color blindnesses
  • Realtime animated MPS Blur

The How

The basic structure of this setup is as follows:

  • Metal Performance Shader
  • AVPlayer
  • AVSampleBufferDisplayLayer (Yes, we can get it to work)
  • CADisplayLink

We setup the AVPlayer as you would normally, with the exception we are not going to add an AVPlayerLayer, we use an AVSampleBufferDisplayLayer

Notes:

  • It's written in Swift 3, I've been using Swift 2 for a while now, the upgrade seemed pretty painless.
  • The Demo could could be wrapped into helper functions to be shared, but I just wanted it written clearly in each demo.
  • There is no buffering code in the demo, I wanted to keep it lean.

Caveats:

  • You have to modify AVSampleBufferDisplayLayer in Xcode to get it to run. I've included a modified version in this repository.
  • You need to run on a device, Metal isn't supported in the simulator
  • NSAllowsArbitraryLoads is set to yes in the Demo to load videos from any url

Resources:

  • Buy Metal By Example by Warren Moore, it's a great book
You can’t perform that action at this time.