Example of use in iOS?

[wehriam]

Hi, thanks for making this library available. Are there any examples of use in an iOS environment, specifically with an iPhone camera?

Thanks!

[krad]

Currently the only example is the pupil video ingest server (https://github.com/krad/pupil)

The docs show how you'd piece together a mp4 writer (similar to a simple AVAssetWriter) http://www.krad.io/morsel/Classes/FragmentedMP4Writer.html#/s:6morsel19FragmentedMP4WriterC6appendy4grip16CompressedSample_p6sample_tF

The one difference is it currently only accepts CompressedSample types, which is just a simple struct with raw uint8 video data and timestamp information.

If I find myself with some time, I'll add an example of how to create local HLS assets on an iPhone/iPad

If you'd like to make an attempt I'd be very happy to receive a PR with any code or documentation help

[mpkupriyanov]

Hi

The one difference is it currently only accepts CompressedSample types, which is just a simple struct with raw uint8 video data and timestamp information.

I don't understand, how i can get CompressedSample from CMSampleBuffer?

I get CMSampleBuffer from:

protocol AVCaptureVideoDataOutputSampleBufferDelegate {
        optional func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection)
}

Can you write an example?

[krad]

Hi there - Please pardon my absence with updates to this project. I've been occupied with work and had to move across the country a couple times over the last couple years.

When I last left off the dependencies were scattered across a couple different repos housed under this account.

CompressedSample can be found in the grip project https://github.com/krad/grip
It's a protocol with two implementations VideoSample and AudioSample

When I was writing this project, Apple was not shipping any of their media libs for server side swift. So types like CMSampleBuffer were not available and had to kinda be mocked out to support writing audio video data over a network and be able to reliably reconstruct images and sound on the server side.

I'm no longer working in the media industry so don't have a lot of bandwidth to maintain everything here full time.
I can assure you though that I was able to use the code across these repos to build a live streaming app, so when in doubt just check out some of the other projects. Pupil is an ingest server that will run on linux and produce fragmented mp4's and HLS playlists that can then be played using either a AVPlayer or via the web using something like video.js or slugline / plainview

CMSampleBuffer is kind of an opaque type used in all sorts of ways across Apple's ecosystem. At its core you just need to be able to extract sample data into a uint8 array and any relevant clock information and flags that may be important to the fmp4 writer. Things like if a packet is a key frame or if its an audio frame....what kind of encoder its using, if the sample data in it is compressed at all, etc etc

That's super important because running a capture session isn't gonna produce h264 and aac data that you can just throw at the fmp4 file writer and have it work. You need to run data captured through the appropriate encoders. Helpers for that are available in https://www.github.com/krad/kubrick

https://github.com/krad/grip/blob/master/Sources/grip/CompressedSample.swift
https://github.com/krad/grip/blob/master/Sources/grip/Packets/VideoSamplePacket.swift
https://github.com/krad/grip/blob/master/Sources/grip/Packets/AudioSamplePacket.swift

Here's a test showing how to construct a VideoSamplePacket https://github.com/krad/grip/blob/master/Tests/gripTests/VideoSampleTests.swift

The videoPayload in that example is just an array of bytes.

If you're retrieving samples from a capture session you need to first compress them using either an h264 or h265 encoder (when I first made these libs h265 wasn't availabile.....h264 is probably easier to get started with as it'll be easier to verify it works as its WAY older)

You can use this encoder https://github.com/krad/kubrick/blob/master/Sources/kubrick/Encoders/H264Encoder.swift

There's also an aac encoder for audio....I suggest starting with a video only stream until you got a working pipeline and are comfortable with whats going on. When you're ready to add audio you can use this https://github.com/krad/kubrick/blob/master/Sources/kubrick/Encoders/AACEncoder.swift

That audio encoder is useful cause I believe it will handle two things:

1. AAC encoder "priming" (Google that)
2. Duplicating audio samples to produce a stereo track from a mono capture device (when the lib was written, iOS only captured mono audio.....trying to playback an fmp4 with mono audio didn't work at the time....so the solution was to dupe the audio for the left and right channels)

As for retrieving the raw uint8 payloads from compressed sample buffers, this should help https://github.com/krad/kubrick/blob/master/Sources/kubrick/Samples/Sample.swift

Apologies for how scattered all this is. I wrote this stuff years ago while trying to build a streaming platform during an unemployment stint in my career. Audio and Video streaming is like SUPER hard and there are a lot of details to get right and not a lot of help out there when trying to run down why something doesn't work.

That being said patches and improvements to the project are very much welcome.

[mpkupriyanov]

Thanks a lot

I think this post will help not only me!