feat: Add support for LiDAR, TrueDepth, External (USB) and Continuity Camera Devices (iOS 17)

[mrousavy]

What

This PR consists of 3 changes:

1. ⚠️ BREAKING CHANGE: Renames CameraDevice.devices to CameraDevice.physicalDevices
2. Adds support for four new Camera Devices on iOS:
   • External Cameras (On iPad, external devices are those that conform to the UVC (USB Video Class) specification)
   • Continuity Cameras (e.g. your Mac's Camera streamed through AirPlay)
   • True Depth Camera (a Camera device streaming both depth data (through Infrared) and colors (YUV))
   • LiDAR Camera (a Camera device streaming both depth data (through a LiDAR sensor) and colors (YUV))
     This is based on feat: New JS API for useCameraDevice and useCameraFormat #1784 which adds a new Camera Devices API that allows us to listen for Camera Device changes (USB camera plugged in/out)
     EDIT: Actually external and continuity camera devices were supported before (after the change in feat: New JS API for useCameraDevice and useCameraFormat #1784), they were just reported as wide-angle cameras. I think I'm gonna leave it at that for now, no need to change that.
3. Expose userPreferredCameraDevice, which is a device the user prefers to use across apps (iOS 17). If not set, use the first available Camera Device (usually id 0)

Blockers

Currently, a device looks like this:

{
  devices: ('wide-angle-camera' | 'ultra-wide-angle-camera' | 'telephoto-camera')[]
  position: 'front' | 'back' | 'external'
}

But the external and continuity cameras don't have the device types. They don't tell me whether they're wide-angle or ultra-wide etc., they just say they're external. Therefore I either need to add a unknown or external type to the devices, which is kinda lame as the lens information should be known (through FOV/focal length). Only the position should be external in that case.
I'll need to investigate this, but unfortunately I don't have a device to test this on. Someone wanna borrow me an iPad?

Changes

Tested on

Related issues

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
react-native-vision-camera	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 21, 2023 2:29pm

[tiste]

Hello @mrousavy, I just tried on an iPad with this update and it works! (I casted a HDMI from my computer with a video capture card)

if #available(iOS 17, *) {
          deviceTypes.append(.external)
      }

[mrousavy]

@tiste wow this is amazing!!! can you do a demo of the Bunny Video being streamed to your iPad? I would love to post this on Twitter 😅

[tiste]

Here it is!
As I said, you can update the code with the iOS17 part above.

RPReplay_Final1695807197.MP4

[tiste]

Just run into issue on eas build with:

+-------------+-------------------------+

|           Build environment           |

+-------------+-------------------------+

| xcode_path  | /Applications/Xcode.app |

| gym_version | 2.213.0                 |

| sdk         | iPhoneOS16.4.sdk        |

+-------------+-------------------------+

(/Users/expo/workingdir/build/app/node_modules/react-native-vision-camera/ios/CameraDevicesManager.swift:81:27)

  79 |     }
  80 |     if #available(iOS 17, *) {
> 81 |       deviceTypes.append(.external)
     |                           ^ type 'AVCaptureDevice.DeviceType' has no member 'external'
  82 |     }
  83 |     // iOS 17 specifics:
  84 |     //  This is only reported if `NSCameraUseExternalDeviceType` is set to true in Info.plist,

Think it's due to iPhoneOS16.4, but don't really know how to change it.

[mrousavy]

Yea, I think EAS doesn't have Xcode 15 yet. That part requires Xcode 15