Migrate to standalone ML Kit and add Image Labeling

[davidgovea]

Summary

Addresses #2908
It's unclear how long the new ML Kit will be in beta, so this might be here a while.

What's done:

• Refactor face/barcode/text detection to ML Kit on both platforms
• Remove Firebase requirement (no google-services.json / GoogleServices.plist needed any more)
• Add on-device Image labeling to library and example app

The example is working well!

What's TODO:

• Provide screenshots
• Update documentation
  • iOS minimum target increased to 10
  • Notes on firebase removal
  • Migration notes
  • Image labeling notes (new subspec on iOS)
• Modularity
  • Test including only some subspecs on iOS. Make sure conditional code blocks are working as intended
  • Consider Android modularity. Currently, it's "all or nothing" for ML Kit. Since most models are packaged locally, it might be desirable to be able to pick&choose between Face/Barcode/Text/Labels. Could use some advice here.
• Object tracking
  • The other new ML Kit feature. Could add the base model functionality
• Custom models
  • Research how custom models are used & provide notes

Test Plan

What's required for testing (prerequisites)?

What are the steps to reproduce (after prerequisites)?

Compatibility

OS	Implemented
iOS	✅
Android	✅

Checklist

• I have tested this on a device and a simulator
• I added the documentation in README.md
• I mentioned this change in CHANGELOG.md
• I updated the typed files (TS and Flow)
• I added a sample use of the API in the example project (example/App.js)

[MSchmidt]

Does the stand alone MLKit run without Google Play Services? This would be great to support more devices. Like current Huawei devices.

[davidgovea]

@MSchmidt yes, it appears so!

I removed the google play services plugin from the mlkit example app (see here), and everything is working nicely!

I'll try to update this PR soon with screenshots & more info, but for now I'm not rushing, since it looks like this repo will wait for the new ML Kit to leave beta before adopting it, and could be a while.

[mneuber]

I would like to request if we can publish a dedicated ml kit standalone beta release. In the migration guide to the ML Standalone Kit is stated:

by delaying the switch to the new ML Kit SDK, you will not benefit from new features and updates. In addition, once you update other components in your app there is a risk you may run into dependencies conflicts.

( https://developers.google.com/ml-kit/migration at the bottom )

I think it would be good to align with their release policy, as we would not be at Day 1, when the old ML Kit gets deprecated and we can start to test and use it in production.

[Lomaas]

Would love that aswell. Beta release

@mneuber

I would like to request if we can publish a dedicated ml kit standalone beta release. In the migration guide to the ML Standalone Kit is stated:

by delaying the switch to the new ML Kit SDK, you will not benefit from new features and updates. In addition, once you update other components in your app there is a risk you may run into dependencies conflicts.

( https://developers.google.com/ml-kit/migration at the bottom )

I think it would be good to align with their release policy, as we would not be at Day 1, when the old ML Kit gets deprecated and we can start to test and use it in production.

Would like that as well!

[mikehardy]

Hey @davidgovea - taking this in a slightly different direction, react-native-firebase is also processing the separation from 'Firebase ML Vision' to a split of 'Firebase ML' (cloud-based inference) and on-device inference in 'Google ML Kit'.

While there is a great deal of overlap between react-native-camera's on-device inference capabilities, there is also a great deal of non-camera functionality in Google ML Kit (text extraction from documents, processing non-camera images, language translation, and running downloaded models or bundled models).

Because of the non-camera features I'm obligated to code up react-native-mlkit to wrap the Google ML Kit APIs, and that means I'll have all sorts of repo infrastructure for testing, distribution etc. Additionally, the (deprecated) @react-native-firebase/ml-vision package had tons of on-device inference functionality that already overlapped with react-native-camera's on-device inference features so I already have a base implementation to just re-home

What I'm thinking is, we should de-duplicate, in the process purging a ton of code out of react-native-camera (perhaps making it more maintainable...) and helping react-native-mlkit because it will have a built in audience of users to keep it moving along (apart from the react-native-firebase users that will migrate to it as well.

What do you think?

If you like the idea then the real migration here in this PR might be to make sure that whatever interfaces react-native-camera needed were easily available in react-native-mlkit so you could depend on it and get what you needed

As a start, it would look something like the APIs that were purged out of react-native-firebase (links to old docs):

Usage https://react-native-firebase-a1v1mzwb1.vercel.app/ml-vision/usage
Reference https://react-native-firebase-a1v1mzwb1.vercel.app/reference/ml-vision

🤔

[davidgovea]

@mikehardy - I agree.

Currently the native-side functionality is:

• on iOS, a UIImage is created from sample buffer and passed to detectors using captureOutput:didOutputSampleBuffer
• on Android, a byte array is passed to the detectors, using a Camera.PreviewCallback

Also, RNCamera throttles detector calls on iOS, but I don't see similar logic on Android. Interestingly, some throttling happens in JS, too. Currently, the throttling is not configurable, and each detector type has its own throttle timer. Perhaps this could be one configurable "frame preview interval" on the native side. Seems like this would be the camera's responsibility, not the mlkit library's

---

The other option is to send everything over the JS bridge, but this has obvious performance downsides.

tfjs-react-native attempts to solve this challenge with native-side preprocessing:
Camera data is resized on native via a WebGL program before being sent across the bridge

---

Just to clarify, you do mean that react-native-mlkit would expose Java / Objective-C interfaces, correct? Like today, gradle/Podfile config could be used to add the additional code/hooks that call MLKit.

Sounds doable!

[mikehardy]

Thanks for thinking it through @davidgovea - I'm replying quickly (mostly to say thanks) prior to thinking through your response, apologies. Quick take is that normally I'd say it's best not to puncture the abstraction and that it should be JS-land, but this is too far to go in an area with data processing and performance needs like this, so I think some sort of native ability will be required for it to not be terrible. How to do it well? That will be the work :-). I'll follow your links and thought process above and see if I can sketch something out to debate

[DiMaNacho]

Please, fix problems and merge it, this PR solves conflicts with 10.x.x version of Firebase if you need to use in your project.

Link of issue:

#3040

[mikehardy]

I was roped in to other firebase-related items (implementing auth.useEmulator, and then doing a forward port for FlutterFire) so haven't had time to push this forward, sorry. If this works as is, I strongly believe "perfect is the enemy of good" and there is no problem with incrementalism - that is, if it's working merge it. Nothing stopping a reorganization later. I will eventually do a react-native-mlkit regardless and I'll do my best at any time to make it useful for use cases like this one

[amrdraz]

@davidgovea we noticed while using it that we no longer get the smile and eye data

we expected

export interface Face {
  faceID?: number;
  bounds: {
    size: Size;
    origin: Point;
  };
  smilingProbability?: number;
  leftEarPosition?: Point;
  rightEarPosition?: Point;
  leftEyePosition?: Point;
  leftEyeOpenProbability?: number;
  rightEyePosition?: Point;
  rightEyeOpenProbability?: number;
  leftCheekPosition?: Point;
  rightCheekPosition?: Point;
  leftMouthPosition?: Point;
  mouthPosition?: Point;
  rightMouthPosition?: Point;
  bottomMouthPosition?: Point;
  noseBasePosition?: Point;
  yawAngle?: number;
  rollAngle?: number;
}

but get

faceID?: number;
  bounds: {
    size: Size;
    origin: Point;
  };
  yawAngle?: number;
  rollAngle?: number;

[davidgovea]

@amrdraz what platform(s)?

And to confirm, you are enabling those modes via props:

        faceDetectionLandmarks={RNCamera.Constants.FaceDetection.Landmarks.all}
        faceDetectionClassifications={RNCamera.Constants.FaceDetection.Classifications.all}

[amrdraz]

@danieldkim we initially had

faceDetectionLandmarks={
                RNCamera.Constants.FaceDetection.Landmarks?.all
              }
              faceDetectionClassifications={
                RNCamera.Constants.FaceDetection.Classifications?.all
              }

this is because we would get sometimes

TypeError: undefined is not an object (evaluating 'RNCamera.Constants.FaceDetection.Landmarks.all')

But evidently it seems that it maybe always undefined (with this MR).

We confirm the issue on IOS, on android it works fine

[davidgovea]

@amrdraz thanks for the report! Just pushed up a fix for that -- it was the conditional-compilation of the constants (forgot to update a firebase ML kit reference).

Tested the example app on iOS -- was able to get the landmarks and classifications as expected now.

---

I have some updates for this PR soon! Pose detection, modular features on android. Seems like folks want this. We'll see if a fork/secondary package is appropriate, but I'm hopeful we can keep things homed here.

[Peretz30]

@davidgovea is this solution work with rectOfInterest prop?

[davidgovea]

@Peretz30 nope. Just took a look, and on Android, that prop is currently implemented only for the non-mlkit barcode scanning task (onBarCodeRead vs onGoogleVisionBarcodesDetected). Didn't dig into iOS.

The process of getting frame preview data to the detectors (and throttling the calls) could use some rework. I've withheld from reorganizing & restructuring during the course of this PR

[stolktech]

I just wanted to share that I really appreciate the work that has been done so far. This is the PR I need (And I'm currently using in my app) to have Firebase Database + Text recognition working in my app.

[stolktech]

Like I said in my previous comment. I'm currently using this PR to test some stuff. I use firebase/database together with this version of RNCamera. I did the changes you mention in the issue on the main repo, but I have trouble building for android.

/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:33: error: package org.reactnative.imagelabeler does not exist
import org.reactnative.imagelabeler.RNImageLabeler;
                                   ^
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/tasks/ImageLabelerAsyncTaskDelegate.java:4: error: package org.reactnative.imagelabeler does not exist
import org.reactnative.imagelabeler.RNImageLabeler;
                                   ^
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:74: error: cannot find symbol
  private RNImageLabeler mImageLabeler;
          ^
  symbol:   class RNImageLabeler
  location: class RNCameraView
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:608: error: cannot find symbol
  public void onImageLabelingError(RNImageLabeler imageLabeler) {
                                   ^
  symbol:   class RNImageLabeler
  location: class RNCameraView
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/tasks/ImageLabelerAsyncTaskDelegate.java:10: error: cannot find symbol
    void onImageLabelingError(RNImageLabeler imageLabeler);
                              ^
  symbol:   class RNImageLabeler
  location: interface ImageLabelerAsyncTaskDelegate
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraViewHelper.java:22: error: package org.reactnative.imagelabeler does not exist
import org.reactnative.imagelabeler.RNImageLabeler;
                                   ^
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraViewHelper.java:352: error: cannot find symbol
  public static void emitImageLabelingErrorEvent(final ViewGroup view, final RNImageLabeler imageLabeler) {
                                                                             ^
  symbol:   class RNImageLabeler
  location: class RNCameraViewHelper
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/events/LabelDetectionErrorEvent.java:11: error: package org.reactnative.imagelabeler does not exist
import org.reactnative.imagelabeler.RNImageLabeler;
                                   ^
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/events/LabelDetectionErrorEvent.java:15: error: cannot find symbol
  private RNImageLabeler mImageLabeler;
          ^
  symbol:   class RNImageLabeler
  location: class LabelDetectionErrorEvent
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/events/LabelDetectionErrorEvent.java:20: error: cannot find symbol
  public static LabelDetectionErrorEvent obtain(int viewTag, RNImageLabeler imageLabeler) {
                                                             ^
  symbol:   class RNImageLabeler
  location: class LabelDetectionErrorEvent
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/events/LabelDetectionErrorEvent.java:29: error: cannot find symbol
  private void init(int viewTag, RNImageLabeler imageLabeler) {
                                 ^
  symbol:   class RNImageLabeler
  location: class LabelDetectionErrorEvent
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:224: error: cannot find symbol
          new ImageLabelerAsyncTask(delegate, mImageLabeler, data, width, height, correctRotation, getResources().getDisplayMetrics().density, getFacing(), getWidth(), getHeight(), mPaddingX, mPaddingY).execute();
              ^
  symbol: class ImageLabelerAsyncTask
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:590: error: cannot find symbol
    mImageLabeler = new RNImageLabeler(mThemedReactContext);
                        ^
  symbol:   class RNImageLabeler
  location: class RNCameraView
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/com/google/android/cameraview/Camera2.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
13 errors

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':react-native-camera:compileGeneralReleaseJavaWithJavac'.
> Compilation failed; see the compiler error output for details.

This is the changes in my app/build.gradle:

    implementation (project(':react-native-camera')) {
        exclude group: "com.google.android.gms"
        exclude group: "com.google.android.gms", module: "play-services-vision"
        exclude group: "com.google.firebase", module: "firebase-ml-vision-face-model"
    }

    implementation platform('com.google.firebase:firebase-bom:26.1.0')
    implementation 'com.google.firebase:firebase-analytics'

    implementation fileTree(dir: "libs", include: ["*.jar"])
    implementation 'com.android.support:multidex:2.0.1'

And ofcourse I'm using this in my package.json:

"@react-native-firebase/app": "^10.1.0",
"@react-native-firebase/database": "^10.1.0",
"react-native-camera": "davidgovea/react-native-camera#standalone-mlkit",

It's probably something obvious for you guys that I'm missing, I just started using react native.

[roffelund]

Great work on this!

What are the plans regarding this? Since 8th of December MLKit is no longer in Beta. Do you need any assistance to complete this or are you planning to extract all the functionality regarding MLKit into a separate package?

[stolktech]

@davidgovea you have any idea what I'm missing in my build script for android?

[zappingseb]

@stolktech

I was able to run it without any build errors with

/android/app/build.grade

android{
    defaultConfig: {
         missingDimensionStrategy 'react-native-camera', 'mlkit'
    }
}

dependencies {
    implementation (project(':react-native-camera')) {
        exclude group: "com.google.android.gms", module: "play-services-vision"
        exclude group: "com.google.android.gms", module: "play-services-vision-common"
        exclude group: "com.google.android.gms", module: "play-services-vision-face-contour-internal"
    }

    implementation platform('com.google.firebase:firebase-bom:26.1.0')
    implementation 'com.google.firebase:firebase-analytics'
    
    implementation fileTree(dir: "libs", include: ["*.jar"])
    ....

package.json

    "@react-native-community/google-signin": "^5.0.0",

    "@react-native-firebase/app": "^10.1.0",
    "@react-native-firebase/auth": "^10.2.0",
    "@react-native-firebase/firestore": "^10.3.1",
    "@react-native-firebase/ml": "^10.1.1",

    "react": "16.13.1",
    "react-native": "0.63.3",

    "react-native-camera": "davidgovea/react-native-camera#standalone-mlkit",

I hope this is useful for you. It also worked out on iOS without making any changes from the react-native-camera documentation

[Peretz30]

Like I said in my previous comment. I'm currently using this PR to test some stuff. I use firebase/database together with this version of RNCamera. I did the changes you mention in the issue on the main repo, but I have trouble building for android.

/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:33: error: package org.reactnative.imagelabeler does not exist
import org.reactnative.imagelabeler.RNImageLabeler;
                                   ^
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/tasks/ImageLabelerAsyncTaskDelegate.java:4: error: package org.reactnative.imagelabeler does not exist
import org.reactnative.imagelabeler.RNImageLabeler;
                                   ^
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:74: error: cannot find symbol
  private RNImageLabeler mImageLabeler;
          ^
  symbol:   class RNImageLabeler
  location: class RNCameraView
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:608: error: cannot find symbol
  public void onImageLabelingError(RNImageLabeler imageLabeler) {
                                   ^
  symbol:   class RNImageLabeler
  location: class RNCameraView
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/tasks/ImageLabelerAsyncTaskDelegate.java:10: error: cannot find symbol
    void onImageLabelingError(RNImageLabeler imageLabeler);
                              ^
  symbol:   class RNImageLabeler
  location: interface ImageLabelerAsyncTaskDelegate
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraViewHelper.java:22: error: package org.reactnative.imagelabeler does not exist
import org.reactnative.imagelabeler.RNImageLabeler;
                                   ^
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraViewHelper.java:352: error: cannot find symbol
  public static void emitImageLabelingErrorEvent(final ViewGroup view, final RNImageLabeler imageLabeler) {
                                                                             ^
  symbol:   class RNImageLabeler
  location: class RNCameraViewHelper
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/events/LabelDetectionErrorEvent.java:11: error: package org.reactnative.imagelabeler does not exist
import org.reactnative.imagelabeler.RNImageLabeler;
                                   ^
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/events/LabelDetectionErrorEvent.java:15: error: cannot find symbol
  private RNImageLabeler mImageLabeler;
          ^
  symbol:   class RNImageLabeler
  location: class LabelDetectionErrorEvent
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/events/LabelDetectionErrorEvent.java:20: error: cannot find symbol
  public static LabelDetectionErrorEvent obtain(int viewTag, RNImageLabeler imageLabeler) {
                                                             ^
  symbol:   class RNImageLabeler
  location: class LabelDetectionErrorEvent
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/events/LabelDetectionErrorEvent.java:29: error: cannot find symbol
  private void init(int viewTag, RNImageLabeler imageLabeler) {
                                 ^
  symbol:   class RNImageLabeler
  location: class LabelDetectionErrorEvent
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:224: error: cannot find symbol
          new ImageLabelerAsyncTask(delegate, mImageLabeler, data, width, height, correctRotation, getResources().getDisplayMetrics().density, getFacing(), getWidth(), getHeight(), mPaddingX, mPaddingY).execute();
              ^
  symbol: class ImageLabelerAsyncTask
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:590: error: cannot find symbol
    mImageLabeler = new RNImageLabeler(mThemedReactContext);
                        ^
  symbol:   class RNImageLabeler
  location: class RNCameraView
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/com/google/android/cameraview/Camera2.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
13 errors

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':react-native-camera:compileGeneralReleaseJavaWithJavac'.
> Compilation failed; see the compiler error output for details.

This is the changes in my app/build.gradle:

    implementation (project(':react-native-camera')) {
        exclude group: "com.google.android.gms"
        exclude group: "com.google.android.gms", module: "play-services-vision"
        exclude group: "com.google.firebase", module: "firebase-ml-vision-face-model"
    }

    implementation platform('com.google.firebase:firebase-bom:26.1.0')
    implementation 'com.google.firebase:firebase-analytics'

    implementation fileTree(dir: "libs", include: ["*.jar"])
    implementation 'com.android.support:multidex:2.0.1'

And ofcourse I'm using this in my package.json:

"@react-native-firebase/app": "^10.1.0",
"@react-native-firebase/database": "^10.1.0",
"react-native-camera": "davidgovea/react-native-camera#standalone-mlkit",

It's probably something obvious for you guys that I'm missing, I just started using react native.

Same here. I am able to run build in debug mode with react-native run-android, but when trying to create apk with ./gradlew assembleRelease there are build errors.

[comvenger-brandon]

I'm unable to build a production iOS app with this change, it throws some errors saying "Undefined symbols for architecture armv7" for the MLKIT. Development version is working fine (or when "Build active architecture only" is set to true).

This page explicitly states, that the MLKIT is not compatible with armv7. https://developers.google.com/ml-kit/migration/ios
Do we need to adjust our build settings? The regular react native config includes armv7 builds and I think there are still many 32bit devices out there.

Can someone please shed some light here?

[mikehardy]

@comvenger-brandon I question this assertion:

I think there are still many 32bit devices out there.

iPhone 5S and newer are 64bit:

https://developer.apple.com/library/archive/documentation/DeviceInformation/Reference/iOSDeviceCompatibility/DeviceCompatibilityMatrix/DeviceCompatibilityMatrix.html

That is apparently 99.5%+ of the market as this table contains 5S but not 5 and lower:
https://deviceatlas.com/blog/most-popular-iphones

Additionally those phones have been officially unsupported for a while:
https://en.wikipedia.org/wiki/List_of_iOS_and_iPadOS_devices#RAM,_RAM_type,_processor,_storage_type_and_highest_supported_iOS_release

It's not fun trimming off groups of users but it appears moot if mlkit doesn't support them, so a build settings adjustment is likely needed. Not sure exactly the setting to change, perhaps moving cocoapods platform directive in Podfile to 12 is sufficient? If not then something about valid arches in a post install hook in podfile or your xcode settings modified directly in xcode gui

[comvenger-brandon]

Ok, looks like you're right. It seemed more to me at first glance. I think most projects are fine with dropping iPhone 5 and older then.

But still, this is something to consider when merging this PR, it should be added to the installation docs.

[zoontek]

This merge request is really important for us, so I will be glad to help if needed. I can test it on multiple devices, review the code, etc 🙂

[nguyen95]

Like I said in my previous comment. I'm currently using this PR to test some stuff. I use firebase/database together with this version of RNCamera. I did the changes you mention in the issue on the main repo, but I have trouble building for android.

/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:33: error: package org.reactnative.imagelabeler does not exist
import org.reactnative.imagelabeler.RNImageLabeler;
                                   ^
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/tasks/ImageLabelerAsyncTaskDelegate.java:4: error: package org.reactnative.imagelabeler does not exist
import org.reactnative.imagelabeler.RNImageLabeler;
                                   ^
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:74: error: cannot find symbol
  private RNImageLabeler mImageLabeler;
          ^
  symbol:   class RNImageLabeler
  location: class RNCameraView
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:608: error: cannot find symbol
  public void onImageLabelingError(RNImageLabeler imageLabeler) {
                                   ^
  symbol:   class RNImageLabeler
  location: class RNCameraView
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/tasks/ImageLabelerAsyncTaskDelegate.java:10: error: cannot find symbol
    void onImageLabelingError(RNImageLabeler imageLabeler);
                              ^
  symbol:   class RNImageLabeler
  location: interface ImageLabelerAsyncTaskDelegate
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraViewHelper.java:22: error: package org.reactnative.imagelabeler does not exist
import org.reactnative.imagelabeler.RNImageLabeler;
                                   ^
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraViewHelper.java:352: error: cannot find symbol
  public static void emitImageLabelingErrorEvent(final ViewGroup view, final RNImageLabeler imageLabeler) {
                                                                             ^
  symbol:   class RNImageLabeler
  location: class RNCameraViewHelper
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/events/LabelDetectionErrorEvent.java:11: error: package org.reactnative.imagelabeler does not exist
import org.reactnative.imagelabeler.RNImageLabeler;
                                   ^
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/events/LabelDetectionErrorEvent.java:15: error: cannot find symbol
  private RNImageLabeler mImageLabeler;
          ^
  symbol:   class RNImageLabeler
  location: class LabelDetectionErrorEvent
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/events/LabelDetectionErrorEvent.java:20: error: cannot find symbol
  public static LabelDetectionErrorEvent obtain(int viewTag, RNImageLabeler imageLabeler) {
                                                             ^
  symbol:   class RNImageLabeler
  location: class LabelDetectionErrorEvent
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/events/LabelDetectionErrorEvent.java:29: error: cannot find symbol
  private void init(int viewTag, RNImageLabeler imageLabeler) {
                                 ^
  symbol:   class RNImageLabeler
  location: class LabelDetectionErrorEvent
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:224: error: cannot find symbol
          new ImageLabelerAsyncTask(delegate, mImageLabeler, data, width, height, correctRotation, getResources().getDisplayMetrics().density, getFacing(), getWidth(), getHeight(), mPaddingX, mPaddingY).execute();
              ^
  symbol: class ImageLabelerAsyncTask
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:590: error: cannot find symbol
    mImageLabeler = new RNImageLabeler(mThemedReactContext);
                        ^
  symbol:   class RNImageLabeler
  location: class RNCameraView
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/com/google/android/cameraview/Camera2.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
13 errors

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':react-native-camera:compileGeneralReleaseJavaWithJavac'.
> Compilation failed; see the compiler error output for details.

This is the changes in my app/build.gradle:

    implementation (project(':react-native-camera')) {
        exclude group: "com.google.android.gms"
        exclude group: "com.google.android.gms", module: "play-services-vision"
        exclude group: "com.google.firebase", module: "firebase-ml-vision-face-model"
    }

    implementation platform('com.google.firebase:firebase-bom:26.1.0')
    implementation 'com.google.firebase:firebase-analytics'

    implementation fileTree(dir: "libs", include: ["*.jar"])
    implementation 'com.android.support:multidex:2.0.1'

And ofcourse I'm using this in my package.json:

"@react-native-firebase/app": "^10.1.0",
"@react-native-firebase/database": "^10.1.0",
"react-native-camera": "davidgovea/react-native-camera#standalone-mlkit",

It's probably something obvious for you guys that I'm missing, I just started using react native.

I think this line resolved my issue:
missingDimensionStrategy 'react-native-camera', 'mlkit'

[zoontek]

@davidgovea It might be a good idea to split the MLKit switch and the ImageLabeler feature in two PRs / releases.

[thomas-blitzm]

Any updates on the progress on this issue @davidgovea ?

[rohitshampur-dev]

Like I said in my previous comment. I'm currently using this PR to test some stuff. I use firebase/database together with this version of RNCamera. I did the changes you mention in the issue on the main repo, but I have trouble building for android.

/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:33: error: package org.reactnative.imagelabeler does not exist
import org.reactnative.imagelabeler.RNImageLabeler;
                                   ^
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/tasks/ImageLabelerAsyncTaskDelegate.java:4: error: package org.reactnative.imagelabeler does not exist
import org.reactnative.imagelabeler.RNImageLabeler;
                                   ^
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:74: error: cannot find symbol
  private RNImageLabeler mImageLabeler;
          ^
  symbol:   class RNImageLabeler
  location: class RNCameraView
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:608: error: cannot find symbol
  public void onImageLabelingError(RNImageLabeler imageLabeler) {
                                   ^
  symbol:   class RNImageLabeler
  location: class RNCameraView
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/tasks/ImageLabelerAsyncTaskDelegate.java:10: error: cannot find symbol
    void onImageLabelingError(RNImageLabeler imageLabeler);
                              ^
  symbol:   class RNImageLabeler
  location: interface ImageLabelerAsyncTaskDelegate
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraViewHelper.java:22: error: package org.reactnative.imagelabeler does not exist
import org.reactnative.imagelabeler.RNImageLabeler;
                                   ^
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraViewHelper.java:352: error: cannot find symbol
  public static void emitImageLabelingErrorEvent(final ViewGroup view, final RNImageLabeler imageLabeler) {
                                                                             ^
  symbol:   class RNImageLabeler
  location: class RNCameraViewHelper
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/events/LabelDetectionErrorEvent.java:11: error: package org.reactnative.imagelabeler does not exist
import org.reactnative.imagelabeler.RNImageLabeler;
                                   ^
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/events/LabelDetectionErrorEvent.java:15: error: cannot find symbol
  private RNImageLabeler mImageLabeler;
          ^
  symbol:   class RNImageLabeler
  location: class LabelDetectionErrorEvent
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/events/LabelDetectionErrorEvent.java:20: error: cannot find symbol
  public static LabelDetectionErrorEvent obtain(int viewTag, RNImageLabeler imageLabeler) {
                                                             ^
  symbol:   class RNImageLabeler
  location: class LabelDetectionErrorEvent
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/events/LabelDetectionErrorEvent.java:29: error: cannot find symbol
  private void init(int viewTag, RNImageLabeler imageLabeler) {
                                 ^
  symbol:   class RNImageLabeler
  location: class LabelDetectionErrorEvent
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:224: error: cannot find symbol
          new ImageLabelerAsyncTask(delegate, mImageLabeler, data, width, height, correctRotation, getResources().getDisplayMetrics().density, getFacing(), getWidth(), getHeight(), mPaddingX, mPaddingY).execute();
              ^
  symbol: class ImageLabelerAsyncTask
/Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/org/reactnative/camera/RNCameraView.java:590: error: cannot find symbol
    mImageLabeler = new RNImageLabeler(mThemedReactContext);
                        ^
  symbol:   class RNImageLabeler
  location: class RNCameraView
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: /Users/user/Documents/repository/AppJournal/node_modules/react-native-camera/android/src/main/java/com/google/android/cameraview/Camera2.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
13 errors

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':react-native-camera:compileGeneralReleaseJavaWithJavac'.
> Compilation failed; see the compiler error output for details.

This is the changes in my app/build.gradle:

    implementation (project(':react-native-camera')) {
        exclude group: "com.google.android.gms"
        exclude group: "com.google.android.gms", module: "play-services-vision"
        exclude group: "com.google.firebase", module: "firebase-ml-vision-face-model"
    }

    implementation platform('com.google.firebase:firebase-bom:26.1.0')
    implementation 'com.google.firebase:firebase-analytics'

    implementation fileTree(dir: "libs", include: ["*.jar"])
    implementation 'com.android.support:multidex:2.0.1'

And ofcourse I'm using this in my package.json:

"@react-native-firebase/app": "^10.1.0",
"@react-native-firebase/database": "^10.1.0",
"react-native-camera": "davidgovea/react-native-camera#standalone-mlkit",

It's probably something obvious for you guys that I'm missing, I just started using react native.

Same here. I am able to run build in debug mode with react-native run-android, but when trying to create apk with ./gradlew assembleRelease there are build errors.

instead of ./gradlew assembleRelease do ./gradlew :app:assembleRelease

[panter4]

I get a weird issue when using this code on IOS:

"Undefined symbols for architecture arm64" for various MLKit objects at build-time

I need face-detection so that is the only subspec that is being selected in my podfile

From what i can tell everything seems installed correctly, Xcode even does code-highlighting for the lines where the issue happens. i tried clearing everything, deintegrating pods, and deleting node_modules several times. the issue persists.

Any ideas as to what could be the issue would be very helpful, i'm stumped.

[davidgovea]

Hi all,

With stalebot closing the upstream issue #2908, I'm also closing this draft PR.

Where I got lost with this PR was trying to achieve conditional compilation of features on Android.
With the age & adoption of this project, and MLKit being highly modular, there were a lot of possible build combinations:

• No MLKit
• Old Firebase MLKit
• Partial or full MLKit combinations (face but no labeller, or barcode only, etc)

On iOS, this is pretty easy to achieve.

On Android, there already exists a variety of different shim classes that get overwritten with a flavor switch.
So, do we need massive flavor combinations? ./gradlew build devDebugMlkitFacerecognitionImagelabelerBarcodescannerObjectDetection? Does that even work? (not certain, but I don't think so)

The "best" solution I came to was mangling the srcDirs in gradle during build to pull in the required code. This could be configured similarly to the 'subspecs' on iOS, via gradle build settings. The source-mangling was verryyy hacky and brittle. Wayyy too fancy to be trusted, and could definitely break with gradle upgrades. Just not the type of cleverness you want in production. Never finished it because it was comprised of 100% antipattern.

So, honestly the most practical solution is probably to clone the mlkit work here, and tear things up to your project's requirements :(

---

That said, I recommend checking out https://github.com/cuvent/react-native-vision-camera if you have specific frame-processing requirements. They are using JS worklets from Reanimated 2 (with modifications), like we had discussed previously. I have not used it yet (not doing much RN these days) but it looks really awesome.

[MarcoScabbiolo]

@davidgovea thanks for the PR and sorry it didn't go through.

We're discussing deprecating react-native-camera in favor of react-native-vision-camera

[birken92]

Ohh you are considering deprecating the project.

React-native-camera version 4 did seem really promising. But so does react-native-vision-camera.

Will work on React-native-camera version 4 stop for now or?

[MarcoScabbiolo]

In reality the v4 code adds the same underlying implementations that react-native-vision already has. But attempting to maintain all of the v3 features migrating and refactoring the existing code is extremely cumbersome, expecially in Android. I was already considering ditching Camera1 and Camera2 and updating the minimum api version to 21 to use only CameraX until I saw how react-native-vision was implemented. It is what I was hoping to achieve and has an even better extensibility/plugin system than the one we thought and implemented.

From a more personal opinion, the react-native-camera code is (especially in Android) a mess. There are a lot of overlapping features and most are not documented. A fresh start will be the best way to go, and vision-camera has an excellent architecture and development pace. We don't need 3 repos (camera, camera-kit, camera-vision), we only need one, and vision has proven to be the best candidate. Camera kit is pretty good as well, don't count it out if you need the core camera features, it has amazing performance, but vision promises much more flexibility and extensibility.