firebase_ml_vision example sensor orientation

[wz2b]

On a Pixel 3XL, the firebase vision example doesn't work very well. With no errors, it doesn't decode barcodes out of the box, and the 'camera preview' example doesn't find anything.

To reproduce this, you just have to build this example and run it on a Google Pixel 3XL.

There is a lot of advice about this problem on stackoverflow and other places. One thing that seems to improve the barcode scanning example is to do this:

    defaultTargetPlatform == TargetPlatform.android
        ? ResolutionPreset.high // was medium
        : ResolutionPreset.low;

I can verify that does indeed improve performance, but I don't think it's doing so the right way. My first clue to this is that if you run "Material Barcode Scanner" and hold the phone in portrait mode, it scans with MUCH better reliability if you hold the bar code sideways. You can watch it perform by looking at the bottom where it says "Hold the bar code closer".

As a second example, if you run the Camera Preview Scanner and put it into barcode mode, it draws boxes around where it thinks it sees an area to decode. Again holding the phone vertically (portrait), it doesn't see much. Turn the barcode sideways, and it detects it with a long, skinny box that's "tall" rather than "wide".

I think something has the orientation wrong. There are a number of references like This that explain why orientation is so confusing. Orientation is really talking about the sensor's orientation, not the image. If the sensor is 90 degrees off, then in landscape you might have to rotate 90 + 90 (or possibly 90 - 90) to get the image upright.

Here's an example of that with the barcode detector selected. If I turn the bottle sideways, it doesn't detect ANYTHING. If I turn it the way shown, you can see that it does.

Now here's where it gets even stranger. It doesn't detect any text at all in ResolutionPreset.medium (the default for android). But if I change that to Resolution.high it does. But only in the correct (horizontal, normal, whatever you want to call it) orientation. In other words, in 'high' resolution, text recognition works fine. On this device, at least, there doesn't seem to be any performance penalty for selecting 'high' resolution.

I can fix all these problems with various hard-coding but that's a terrible way to make something that should, in theory, work everywhere. And like I said, these kinds of problems haunted me back in Kotlin as well. But in the context of this project, you have a great example that mostly works out-of-the-box, so i'm wondering if, rather than wait for firebase to fix these issues, we can just make this example more reliable.

Thoughts?

dependencies:
  flutter:
    sdk: flutter
  firebase_core: ^0.4.0+9
  firebase_ml_vision: ^0.9.4
  firebase_analytics: ^5.0.2
  firebase_auth: ^0.14.0+5
  cloud_firestore: ^0.12.9+5
  camera: ^0.5.8+2      /* I tried several different versions with same results */
  image_picker: '0.6.7+2'
  path_provider: '1.6.10'
  cupertino_icons: ^0.1.3

[√] Flutter (Channel stable, v1.17.1, on Microsoft Windows [Version 10.0.19041.329], locale en-US)

[√] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
[√] Android Studio (version 3.6)
[√] IntelliJ IDEA Ultimate Edition (version 2019.3)
[√] VS Code (version 1.45.1)
[√] Connected device (1 available)

[iapicca]

Hi @wz2b
you are making a good point,
honestly I'm not sure that this is caused by flutterfire plugin or the native plugins that builds on,
(also I believe flutter doesn't use camera2 to keep compatibility with sdk16)

Nonetheless I cannot be sure of the opposite.

For now I'd keep the issue open.
Thank you

[wz2b]

Yeah, agreed - I don't think it's caused by flutterfire, either. I think, though, that if I can find a workaround in flutterfire that might be faster than getting it fixed in Firebase itself or, even worse, Android. I found some sample code that actually tweaks the settings based on a static list of quirky phones they know about. Blech.

…

On Fri, Jun 19, 2020 at 9:57 AM Francesco Iapicca ***@***.***> wrote: Hi @wz2b <https://github.com/wz2b> you are making a good point, honestly I'm not sure that this is caused by flutterfire plugin or the native plugins that builds on, (also I believe flutter doesn't use camera2 to keep compatibility with sdk16) Nonetheless I cannot be sure of the opposite. For now I'd keep the issue open. Thank you — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2793 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALTMOJL33Q5VJZTE7SDU3DRXNVEVANCNFSM4OCBPHKQ> .

[wz2b]

Firebase Vision documentation strongly suggests you use YUV_420_888 as the image format. I implemented this once using the GPU and android Render scripts. What I did was took the camera image (RGBA format) and converted it to YUV in a renderscript. It worked well, performance was good, and it improved recognition quite a bit.

The image created in this example is converted to a FirebaseVisionImage using

      FirebaseVisionImage.fromBytes(
        _concatenatePlanes(image.planes),
        _buildMetaData(image, _rotationIntToImageRotation(imageRotation)),
      )

I stopped to inspect there to see what that format is and it is returned as:

So it definitely is YUV420. The detected sensor orientation is 90, and its notion of resolution is 720h x 1280w. I'm holding the phone vertically, so when you apply a 90 degree orientation, you get a resolution that seems right. That image is eventually passed to BarcodeVisionDetector.detectInImage().

One thing I wonder is if detectInImage() genuinely pays attention to the rotation flag in FirebaseVisionImageMetadata. I think when I was using Vision in kotlin I tried rotating it myself but I don't remember the results.

It looks like this code here sends the entire image to the decoder. That's a little misleading - there's a guide box that makes you think that's where it's scanning, but I don't think that's true, is it? If I'm right, then this might be worth looking at, as part of this flutter example. Cropping isn't that expensive an operation compared to decoding. That's more true for character/text recognition (barcode scanning is pretty fast).

UPDATE
According to this StackOverflow the vision library does indeed take rotation into account. So I'm still trying to sort out why it consistently decodes better when the phone is rotated landscape (the wrong way).

[aakashbapna]

I also found that barcode scanning works in High resolution (1280x720) but this results in frequent GCs on old android phones. I have a fork of firebase_ml_vision which uses the newer google ml-kit on device lib and the problem exists there as well.
The ml-kit Vision-quickstart samples written in Java/Kotlin (but use older camera api) work perfectly fine with 720x480 (medium) resolution. I am not able to figure out what's wrong in Flutter land with medium resolution.

Here there is a talk of incompatibility with some devices and certain Camera2 versions- https://developers.google.com/ml-kit/known-issues

[Salakar]

Hey, 👋, the firebase_ml_vision package is now discontinued since its APIs have been deprecated and removed from the Android & iOS Firebase SDKs.

I'd recommend switching to the alternatives now;

• for on-device vision APIs; switch to Google's standalone ML Kit library via google_ml_kit
• for cloud vision APIs; the recommended approach now is to use Firebase
  Authentication and Functions, which gives you a managed, serverless gateway to the Google Cloud Vision APIs. For an example Functions project see the vision-annotate-images sample project. You'd deploy this project for example and then call the cloud function from your app with your image data to get the annotation result.

Apologies for any inconvenience here and best of luck switching over the the new APIs. firebase_ml_custom is not affected by this deprecation.

Thanks