3.0.0 (2023-09-01)

✨ VisionCamera V3!!!! ✨

This is the third major version for VisionCamera, VisionCamera V3, which features a full codebase rewrite on Android, and a huge refactor on iOS to make it more stable, more flexible, and more performant than ever!

Note

If you want to keep using V2, I plan to provide limited support to V2 on the V2 branch here.

Journey

VisionCamera V3 has been an intense journey for me, I spent over 700 hours in total to build VisionCamera V3 and make it as fast and powerful as possible. Lots of research went into this, and writing a custom OpenGL GPU pipeline in C++ is far from easy - there ain't no documentation about this online at all!

Here's the original V3 issue/discussion board: #1376

If you appreciate what I'm doing in VisionCamera, please 💖 consider sponsoring me on GitHub 💖 or 🍪 buy me a Ko-Fi 🍪 to show your support. Thank you!

These are some of the major features:

🤖 1. Full Android Camera2 rewrite

On Android, there's three APIs for using the Camera: Camera1 (deprecated), Camera2 and CameraX. Camera2 is known for being insanely hard to use, so Google built CameraX, a library which uses Camera2 under the hood but significantly simplifies it to make it easier to use.
This sounds great at first, so I used CameraX for VisionCamera V1 and V2. Unfortunately due to their simplifications (and it being immature), a lot of features that worked on iOS were simply broken or not working on Android.

In V3 I now rewrote the entire Android codebase from CameraX to the lower-level Camera2 library, which allows for many new great features:

Multi-cams ("fish-eye" zoom) now work on Android! 🐟
Startup time has been significantly improved and the overall performance is much better! 🔥
Device Formats now finally show the correct videoWidth/videoHeight and photoWidth/photoHeight sizes! ✅
60 FPS Cameras now work! 💨
Greatly increased stability overall throughout the Camera 💪
Build setup got much simpler which fixed hundreds of issues! Not even kidding, this fixed ~130 issues here since we no longer depend on androidx.camera:+ 🥇
Photo capture got much faster and mirroring images is much more performant 🔥

⚠️ Warning

Because this is a full rewrite to a much lower-level library on Android, there might be some things that broke. Please make sure to report an issue if you spot such things.
Currently, these are the things that are not yet working on Android:

❌ Flash (this requires a custom precapture/auto-exposure sequence, I'll work on that soon)
❌ Orientation (this requires a lot of testing and figuring out how orientation has to be handled, some devices like Samsung work differently here)
❌ Focus (focus works, but the coordinate system is wrong. This requires Orientation to work first)

📹 2. Custom GPU OpenGL Video Pipeline

On Android I built a custom GPU OpenGL video pipeline in C++ that will handle the rendering of the input Frames to multiple output Surfaces (currently only Video Recordings and Frame Processing). This is roughly how it works:

Camera-->GL_TEXTURE_EXTERNAL_OES;
  GL_TEXTURE_EXTERNAL_OES -- PassThroughShader -->FP[Frame Processor Output EGLSurface];
  GL_TEXTURE_EXTERNAL_OES -- PassThroughShader -->VR[Video Recorder Output EGLSurface];

The FP and VR output surfaces can be swapped at any point, meaning adding or removing a Frame Processor is much more performant. Also, this implicitly handles resizing the buffers to match the output dimensions fully automatically on the GPU, making it insanely fast.

The main benefit of this pipeline is that we no longer need to attach two outputs to the Camera but only one - the OpenGL VideoPipeline itself. This introduces two new features on Android:

The supportsParallelProcessing prop is gone since this is now always supported! 🔀🤩
Flipping the Camera now works on Android! 🔄🤩

📷 3. New Camera Features

The pixelFormat prop has been added which allows you whether to stream rgb, yuv or native frames in a Frame Processor.
enableShutterSound has been added to takePhoto() to play or mute the sound on photo capture
New Frame Processor features:
- Frames can be converted to a cached byte array using frame.toByteArray(). This is pretty efficient and can be used to process the raw pixels using libraries like react-native-fast-tflite.
- Frames now expose their pixelFormat (either yuv, rgb or native)
- Frames now expose their orientation
- Frames now expose isMirrored
- Frames now expose a timestamp
Camera Devices now expose their sensorOrientation (the orientation that you have to rotate by to get to the device neutral portrait orientation)
Camera Devices expose their hardwareLevel
Videos can now be recorded either in h264 or h265 (HEVC) 🔥
Use minFps/maxFps in favour of frameRateRanges

🧵 4. New Frame Processor Engine (react-native-worklets-core)

Frame Processors now use react-native-worklets-core in favor of react-native-reanimated. With this refactor, there's a few changes:

You can now use JSI HostObjects and HostFunctions from outside in the Frame Processor. For example, using a JSI library like react-native-tflite is now seamlessly possible:
```
// `model` is a C++ JSI HostObject
const model = useTensorflowModel(require('assets/face-detector.tflite'))

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'

  const pixels = frame.toArrayBuffer()
  // `model` can be used in this 'worklet' without copying anything!
  const faces = model.run(pixels)
}, [model])
```
This makes it much easier for general purpose processing libraries to be used inside Frame Processors. For this example, TFLite can run any .tflite model with your Camera, all from JS while still being powered by C++/GPU.
The build setup got a lot simpler since we no longer need to extract reanimated's .aar. This should fix a ton of build errors.
Frame Processors are now fully synchronous

New API to run something asynchronous: runAsync(..):

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log('New Frame')

  runAsync(frame, () => {
    'worklet'
    const faces = detectFaces(frame)
    const face = [faces0]
    console.log(`Detected a new face: ${face}`)
  })
})

New Frames can stream in ('New Frame' being logged) while the async context is still executing detectFaces fully in parallel.

New API to run something at a throttled FPS rate: runAtTargetFps(.., fps):

const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  console.log('New Frame')

  runAtTargetFps(5, () => {
    'worklet'
    const faces = detectFaces(frame)
    console.log(`Detected a new face: ${faces[0]}`)
  })
})

In this case, the face detector will only be called 5 times per second.

🔧 5. New Frame Processor Plugin API

Frame Processor Plugins are now object oriented and can be initialized from JS with custom options. This will allow you to pass options like which model to use, fast or accurate, to a native FP plugin like a face-detection or pose-detection algorithm.

Old syntax:

export function examplePlugin(frame: Frame) {
  'worklet'
  return VisionCameraPlugins.__examplePlugin(frame)
}

New syntax:

const plugin = VisionCameraProxy.getFrameProcessorPlugin('example_plugin')

export function examplePlugin(frame: Frame) {
  'worklet'
  return plugin.call(frame)
}

And the getFrameProcessorPlugin can also accept options, which is a NSDictionary on iOS in the init: call and a Dictionary<> on Android in the constructor.

See the ExampleFrameProcessorPlugin.m/ExampleFrameProcessorPlugin.java for the native changes.

🎨 Skia

If you followed the V3 journey you might know that originally I planned to add Skia support for VisionCamera, allowing you to draw onto a Frame in realtime.

See more details

This was possible until VisionCamera V3 RC.9 with an amazingly simple API:

const paint = Skia.Paint()
paint.setColor('red')

const frameProcessor = useSkiaFrameProcessor((frame) => {
  'worklet'
  const faces = detectFaces(frame)
  faces.forEach((face) => {
    const rect = Skia.Rect(face.x, face.y, face.width, face.height)
    frame.drawRect(face.rectangle, paint)
  })
}, [paint])

Or, to implement color filters (VHS filter, sepia, beauty, invert colors, ...) you could simply use Skia Shaders:

const INVERTED_COLORS_SHADER = `
uniform shader image;

half4 main(vec2 pos) {
  vec4 color = image.eval(pos);
  return vec4(1.0 - color.rgb, 1.0);
}
`

const imageFilter = Skia.ImageFilter.MakeRuntimeShader(INVERTED_COLORS_SHADER)
const paint = Skia.Paint()
paint.setImageFilter(imageFilter)

const frameProcessor = useSkiaFrameProcessor((frame) => {
  'worklet'
  frame.render(paint)
}, [])

...and the resulting texture that was rendered would also be written to a video or photo file if you started capturing.

Buuuuuuut I decided to remove Skia support from VisionCamera, as the codebase just got way to complex for me to maintain the two pipelines (one skia and one without skia). See this PR for more information on that: #1740 (this even includes the entire code for that lol)

If you/your business wants this, reach out to me/us through our website margelo.io and we can build a customized Camera solution for you - it works, it's just not suitable for the VisionCamera repo, I want to keep that lean. On our website we even have a demo with a custom solution for one of our clients, Stori, which implements realtime face filters just like on Snapchat.

Full Changelog

Here's the full changelog:

✨ Features

Full Android rewrite from CameraX to Camera2 🤯
Add pixelFormat property to Camera (df5718d)
Create C++/OpenGL-based Video Pipeline for more efficient Recording and Frame Processing (#1721) (ea3686c), closes #1727
Replace *NativeMap and *NativeArray with Map<K,V> and List<T> for faster JSI -> JNI calls (#1720) (dfb86e1)
Skia for Android (#1731) (a7c137d), closes #1735 #1736
Use ImageReader and use YUV Image Buffers in Skia Context (#1689) (d38ba59)
Video Recording with Camera2 (#1691) (a81bbbc)
Replace Reanimated with RN Worklets (#1468) (a0590dc)
Sync Frame Processors (plus runAsync and runAtTargetFps) (#1472) (30b5615)
Add enableShutterSound prop to takePhoto() 🔊 (#1702) (a46839a)
Allow build without Skia or Frame Processors (#1710) (862e05b)
Full Android rewrite (CameraX -> Camera2) (#1674) (37a3548), closes #1689 #1691
Implement enableZoomGesture on Android (efe6556)
Support focus() on Android (#1713) (23af74a)
Use ByteBuffer for much faster toArrayBuffer() ⚡ (521d7c8)
Draw onto Frame as if it was a Skia Canvas (#1479) (12f850c), closes #1487
Add fpsGraph prop to show a debug view of the current FPS the Camera is drawing at (#1479
Add previewType prop to switch between native OS preview and the Skia Canvas preview view (#1479
Add toByteArray(), orientation, isMirrored and timestamp to Frame (#1487)
Better Native Module Error Detection (#1515) (f791c6b)
Make Frame Processor Plugins object-oriented on iOS as well (#1496) (622d383)
New CameraDevice + CameraFormat detection using CameraX (#1495) (0d83a13)
Add RN 0.72 support (namespace in build.gradle) (7ae15af)
Better Native Module Error Detection (#1515) (f791c6b)
Complete iOS Codebase rewrite (#1647) (375e894)
Expose unified VisionCameraProxy object, make FrameProcessorPlugins object-oriented (#1660) (44ed42d)
Make Frame Processor Plugins object-oriented on iOS as well (#1496) (622d383)
New CameraDevice + CameraFormat detection using CameraX (#1495) (0d83a13)
Rewrite Android C++ part (VisionCameraProxy + JFrame) (#1661) (86dd703)
Use clang-format to keep the C++ codebase clean (#1741) (03b57a7)

💨 Performance Improvements

Improve pixelFormat and add CameraDevice.sensorOrientation (#1729) (0e9f1ca)

🐛 Bug Fixes

Fix CI for V3 (#1475) (f0ea181)
Prevent possible crash in log error handler (222ded4)
Print correct error in build.gradle (6825c1f)
Add missing <regex> header (0635d4a)
Fix global.FrameProcessorPlugins TS error (1f7a2e0)
Make runAsync run truly async by dropping new Frames while executing (3ea5dfa)
Catch device == null error (f227a3e)
Fix runAtTargetFps for multiple invocations per FP (af4e366)
Fix Android namespace (cf19ff2)
Fix missing Camera error (38749fb)
Fix Skia build (#1582) (f03dde7)
ignore duplicate .so libraries in package (ad5d64b)
Improve C++ safety by attaching Cache Invalidator to jsi::Runtime's lifecycle (#1488) (0c3cd66)
Remove RN Skia dependency on Android for now (61f19df)
Remove temp-patch for REA plugin (6deb686)
Set min iOS version to 12.4 (bb7ab30)
Simplify Podfile (0111f2e)
Use global.expo.modules for JSI expo modules (a1af891)
Catch device == null error (f227a3e)
Downgrade Gradle to 7.5.1 (66c6481)
Fix runAtTargetFps for multiple invocations per FP (af4e366)
Fix Android namespace (cf19ff2)
Fix FP Plugin init call (c2096a4)
Fix missing Camera error (38749fb)
Fix namespace (deeb8db)
Fix Skia build (#1582) (f03dde7)
Fix Worklets imports (5fb594c)
ignore duplicate .so libraries in package (ad5d64b)
Improve C++ safety by attaching Cache Invalidator to jsi::Runtime's lifecycle (#1488) (0c3cd66)
Move implementation for FrameProcessorPlugin to .m (bc9c157)
pragma once (127656b)
Remove iOS 13 guard (7c721cc)
Remove RN Skia dependency on Android for now (61f19df)
Remove temp-patch for REA plugin (6deb686)
Set min iOS version to 12.4 (bb7ab30)
Simplify Podfile (0111f2e)
Use global.expo.modules for JSI expo modules (a1af891)
Use correct Gradle version (0139324)
Don't enable Stream Use Case if it's not 100% supported (224b44f)
Fix make_shared not working on FrameHostObject (1197df7)
Fix Metro Config (46c6f28)
Fix recreate when outputs change (71bc242)
Fix various Android crashes by using Stable CameraX version (#1665) (3b3515c)
Keep Outputs (9829dc4)
Return not-determined on Android (debe751)
Run View Finder on UI Thread (1485910)
Use Orientation (c7e4756)
Use AUDIO_BIT_RATE of 16 * 44,1Khz (18de121)
Use Snapshot Template for speed capture prio (17e1bc7)
Add GrMTLHandle import (390f48d)
Clean up codebase (e1b0408)
Don't error if session closes (1f12192)
Fix JByteBuffer (3a0d7b3)
Fix make_shared not working on FrameHostObject (1197df7)
Fix circular dependency in FrameProcessorPlugins.ts (c88605e)
Fix various Android crashes by using Stable CameraX version (#1665) (3b3515c)
Improve build.gradle (07ba0e1)
Improve performance of zoom, fps, hdr and format selection (4b009a0)
Remove unused constructor (94d9b69)
Remove unused imports (617c560)
Use default HDR if not applicable (0a51980)
Use direct ByteBuffer (86468e3)
Error on disabled Frame Processors (d275812)
Export pixelFormat property on iOS (dfee3b1)
Fix node_modules/ directory detection (66c012f)
Lock PixelBuffer for read in toArrayBuffer() (e036b31)
Fix scene mode crash on some emulators (6bbb44d)
Make Skia and Worklets optional in package.json (#1728) (3b04757)

📚 Documentation

Add fancy new Banner ✨ (65aa9b4)
New README (#1744) (2a5c333)
Update CONTRIBUTING (290a20f)
fix wrong comment (533c32a)

🔧 Codebase

Use clang-format to keep the C++ codebase clean (#1741) (03b57a7)

Navigation Menu

Release 3.0.0 ✨

Choose a tag to compare

Sorry, something went wrong.