💨 Make image flipping faster

[mrousavy]

Type

• Enhancement for an existing feature
• An entirely new feature

What

Images from the front camera (selfie) are automatically mirrored (flipped horizontally). The flip function (introduced in #129) works by converting the byte[] to a Bitmap to perform a matrix transformation, then compresses it back into the byte[].

https://github.com/cuvent/react-native-vision-camera/blob/f57714747a2587916c4e7c47bfe2586b1e8966ff/android/src/main/java/com/mrousavy/camera/utils/ImageProxy.save.kt#L44-L52

This byte[] -> Bitmap -> byte[] conversion is a bit of an overhead I'd like to eliminate, even if it only takes 24ms to execute on my Huawai P10. Here are the full benchmarks:

Back camera (no flipping):

2021-05-03 18:57:54.478 31448-31448/com.mrousavy.camera.example I/CameraView.performance: Session configured in 200 ms! Camera: androidx.camera.lifecycle.LifecycleCamera@c475d4
2021-05-03 18:58:17.609 31448-31729/com.mrousavy.camera.example I/CameraView.performance: Finished creating temp file in 1ms
2021-05-03 18:58:18.028 31448-31448/com.mrousavy.camera.example I/CameraView.performance: Finished image capture in 413ms
2021-05-03 18:58:18.033 31448-31729/com.mrousavy.camera.example I/CameraView.performance: Finished image saving in 3ms
2021-05-03 18:58:18.035 31448-31448/com.mrousavy.camera.example I/CameraView.performance: Finished function execution in 444ms

Front camera (flipping):

2021-05-03 18:58:50.155 31448-31448/com.mrousavy.camera.example I/CameraView.performance: Session configured in 129 ms! Camera: androidx.camera.lifecycle.LifecycleCamera@4e5ccc6
2021-05-03 18:58:52.032 31448-31729/com.mrousavy.camera.example I/CameraView.performance: Finished creating temp file in 2ms
2021-05-03 18:58:52.345 31448-31448/com.mrousavy.camera.example I/CameraView.performance: Finished image capture in 314ms
2021-05-03 18:58:52.379 31448-31729/com.mrousavy.camera.example I/CameraView.performance: Flipping Image took 31 ms.
2021-05-03 18:58:52.380 31448-31729/com.mrousavy.camera.example I/CameraView.performance: Finished image saving in 33ms
2021-05-03 18:58:52.381 31448-31448/com.mrousavy.camera.example I/CameraView.performance: Finished function execution in 353ms

Alternatives/Workarounds

I have tried to toggle some flags in the byte[] in-memory, hoping I could flip the relevant Exif flag. I have discovered that the AndroidX ExifInterface does not support writing to in-memory files (why??), so I tried to roll my own approach:

https://github.com/cuvent/react-native-vision-camera/blob/f57714747a2587916c4e7c47bfe2586b1e8966ff/android/src/main/java/com/mrousavy/camera/utils/ImageProxy.save.kt#L19-L41

that unfortunately doesn't work and returns a black image. Maybe this can be of help?

On iOS this can be achieved with no overhead at all by directly mirroring the output connection inside the Camera pipeline, unfortunately CameraX does not provide this functionality.