iOS camera api all formats are unknown

[SasLuca]

I've been trying the new Camera api on a real iPhone 13 device and while I got SDL to get a list of camera devices, it seems like all the available formats of all the available camera are unsupported.

Specifically CoreMediaFormatToSDL always returns SDL_PIXELFORMAT_UNKNOWN.

Is there any undocumented setting or something that I might be missing on causing this issue or is it just a limitation of the API at the moment?

[slouken]

It's probably just an oversight in CoreMediaFormatToSDL(), please feel free to extend it and submit a PR.

[SasLuca]

I did some light research on this and the issue isn't so much with CoreMediaFormatToSDL but with how formats are queried and which ones are supported.

The SDL camera code for coremedia uses CMFormatDescriptionGetMediaSubType but it seems that in order to get all formats a different method must be used as per this SO post, and even then it is not clear if the formats will be supported by SDL.

I will implement the method for getting formats that is suggested in that SO post and see if the formats will then match. If they don't however I am not quite sure how to proceed, would it mean that SDL needs to add support for more formats?

[slouken]

I will implement the method for getting formats that is suggested in that SO post and see if the formats will then match. If they don't however I am not quite sure how to proceed, would it mean that SDL needs to add support for more formats?

It's possible, yes, but it's likely that SDL already supports the common formats used by the camera.

[SasLuca]

It seems like that SO post about CoreMediaIO was a bad lead, as CoreMediaIO is for macos only.

I instead tried to print all the formats the cameras on my iPhone 13 support, it seems like they are part of another enum than what SDL uses. SDL is looking for formats in the CMPixelFormatType, but that list is apparently not exhaustive. A longer list of formats can be found in CVPixelBuffer.h and the formats in this list do indeed match the ones the cameras support on my iPhone 13.

In particular x420, 420v and 420f seem very common. These format codes translate to the following formats:

kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange = '420v'
kCVPixelFormatType_420YpCbCr8BiPlanarFullRange  = '420f'
kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange = 'x420'

I think I could modify the SDL code to use this header and values instead, but from there on I am sadly not quite knowledgeable. Does SDL support these formats? Is there a way for SDL to support this formats?

Not sure what the implications are here and what needs to be done so any help would be appreciated, I would really love to use the Camera API for my mobile game.

[slouken]

Yes, SDL does support these formats. The difference between kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange and kCVPixelFormatType_420YpCbCr8BiPlanarFullRange is not represented in the SDL pixel format, but instead in other metadata for the surface. I'll have to look at this later, but for now you can just return SDL_PIXELFORMAT_NV12 for both of them and it should work.

[SasLuca]

Awesome, I will modify the code on my side for now and see if I can get the camera stuff to work. Thanks a lot.

[slouken]

You're welcome!

[ghost]

This is related to #9610

I was investigating this earlier today and found that adding CASE(kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, SDL_PIXELFORMAT_NV12); to CoreMediaFormatToSDL does not work as expected. In my case, the device is recognized, but the output frames are not displayed correctly.

While debugging I found that the built-in camera on my M1 Macbook Air reports the pixel format kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange but actually outputs frames as kCMPixelFormat_422YpCbCr8_yuvs:

In COREMEDIA_OpenDevice:

(lldb) p (uint32_t)CMFormatDescriptionGetMediaSubType(spec_format.formatDescription)
(uint32_t) 875704438 // '420v'

In COREMEDIA_AcquireFrame:

(lldb) p (uint32_t)CVPixelBufferGetPixelFormatType(image)
(uint32_t) 846624121 // '2vuy'

I had to add CASE(kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, SDL_PIXELFORMAT_UYVY) to get testcamera to work, which doesn't seem right at all.

[SasLuca]

@slouken I tried your suggestion of using SDL_PIXELFORMAT_NV12 for those camera formats, and while it did work at getting past camera device collection, during rendering I would be getting EXC_BAD_ACCESS thrown from this line in COREMEDIA_AcquireFrame.

I then tried @hkva's suggestion of mapping kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange to SDL_PIXELFORMAT_UYVY. This still resulted in an EXC_BAD_ACCESS, but only once, and after that it would actually render! Sadly the rendering was corrupted, as shown in this image.

I then tried updating SDL, since I last got it from the main branch 2 weeks ago or so, and now I just get EXC_BAD_ACCESS on that same line non stop regardless of the format.

Any help regarding what I could try next to debug or fix this would be appreciated, I really want to use the Camera API so I would like to contribute in any way I can, tho sadly I am not very well educated in this particular area.

Some logs that stand out to me and I'll investigate now are INFO: CAMERA: App wanted [(0x0) fmt=SDL_PIXELFORMAT_UNKNOWN interval=1000/1], chose [(4032x3024) fmt=SDL_PIXELFORMAT_UYVY interval=1/1] and Thread Performance Checker: -[AVCaptureSession startRunning] should be called from background thread. Calling it on the main thread can lead to UI unresponsiveness

Here is a picture of the EXC_BAD_ACCESS I am getting. I am also logging with DEBUG_CAMERA on.

[ghost]

@SasLuca I was able to get my integrated webcam working with this change, maybe you'll have some luck?

https://github.com/hkva-fork/SDL/commit/b8ef9ffa41a591a8c6943e348d840d415241973a

0001-camera-Support-CoreMedia-devices-with-NV12-pixel-for.patch

I am seeing the bad access as well. I can repro by running testcamera on my laptop and selecting my iPhone camera as the source.

[SasLuca]

Ill try your fork and let you know if it works, thanks a lot.

[SasLuca]

@hkva tried the wrong branch initially, but I did try 9610-coremedia-camera-nv12 now and still getting bad accesses in that place sadly.

[slouken]

Fixed, thanks!

[SasLuca]

Awesome, will be updating and trying it again soon!