[firebase_ml_vision] Only works on ResolutionPreset.high CameraController #1518

CoreyCole · 2019-11-29T23:25:27Z

Describe the bug
I'm building an android app using flutter that has a QR code scanning feature. I've broken out the QR code logic into a minimum reproduction case here.

Oddly enough, the QR scanning only works on my Samsung devices. My Google Pixel XL and Oneplus 6 both do not detect anything when scanning a QR code using google/firebase ml vision barcode scanning model.

To Reproduce
Steps to reproduce the behavior

clone/build my minimum repro repo
point the app's camera preview at a QR code

Key code locations in the repository are:

android/app/build.gradle where I include the barcode model api:

api 'com.google.firebase:firebase-ml-vision-barcode-model:16.0.2'

lib/src/bloc/services/qr_service.dart where I run the barcode detection on the image:

_processImage(CameraImage image) async {
    if (!_alreadyCheckingImage && !_foundRoomId) {
        _alreadyCheckingImage = true;
        try {
            final barcodes = await barcodeDetector.detectInImage(
                FirebaseVisionImage.fromBytes(
                    _concatenatePlanes(image.planes),
                    FirebaseVisionImageMetadata(
                        rawFormat: image.format.raw,
                        size: Size(image.width.toDouble(), image.height.toDouble()),
                        rotation: ImageRotation.rotation0,
                        planeData: image.planes
                            .map((plane) => FirebaseVisionImagePlaneMetadata(
                                    bytesPerRow: plane.bytesPerRow,
                                    height: plane.height,
                                    width: plane.width,
                                ),
                            )
                            .toList(),
                    ),
                ),
            );
            if (barcodes != null && barcodes.length > 0) {
                try {
                    print('\n~~~');
                    print(barcodes.first.toString());
                    print(barcodes.first.displayValue);
                    print(barcodes.first.valueType);
                    print(barcodes.first.rawValue);
                    print('~~~\n');
                    final barcode = barcodes.first;
                    print(barcode.rawValue);
                    qrResult.sink.add(barcode.rawValue);
                    _foundRoomId = true;
                } catch (err, stack) {
                    print('$err\n$stack');
                }
            }
        } catch (err, stack) {
            debugPrint('$err, $stack');
        }
        _alreadyCheckingImage = false;
    }
}

Uint8List _concatenatePlanes(List<Plane> planes) {
    final WriteBuffer allBytes = WriteBuffer();
    planes.forEach((plane) => allBytes.putUint8List(plane.bytes));
    return allBytes.done().buffer.asUint8List();
}

I'm hoping I'm doing something wrong. But it is very weird that this minimum repro works flawlessly on my Samsung S8, Samsung J7, and Samsung S10+ while it does not work on my Oneplus 6 (android 10) and it does not work on my Google Pixel XL (android 9).

Additional context
Dependency Versions

firebase_core: ^0.4.2+1
firebase_ml_vision: 0.9.3+3
camera: 0.5.7
rxdart: 0.22.4
device_info: 0.4.1

Device List

SM J700H      • 52033d7fe8448323   • android-arm   • Android 6.0.1 (API 23)
SM G950U1     • 988939444c58415246 • android-arm64 • Android 9 (API 28)
SM G975F      • R58M32WE95N        • android-arm64 • Android 9 (API 28)
Pixel XL      • HT76E0201647       • android-arm64 • Android 9 (API 28)
ONEPLUS A6000 • 4241cff9           • android-arm64 • Android 10 (API 29)

Working on these devices

# samsung J7
I/flutter (21900): {
I/flutter (21900):   "board": "universal7580",
I/flutter (21900):   "bootloader": "J700HXXS3BRL3",
I/flutter (21900):   "brand": "samsung",
I/flutter (21900):   "device": "j7e3g",
I/flutter (21900):   "hardware": "samsungexynos7580",
I/flutter (21900):   "host": "SWDH4623",
I/flutter (21900):   "isPhysicalDevice": true,
I/flutter (21900):   "manufacturer": "samsung",
I/flutter (21900):   "model": "SM-J700H",
I/flutter (21900):   "product": "j7e3gxx",
I/flutter (21900):   "tags": "release-keys",
I/flutter (21900):   "type": "user",
I/flutter (21900):   "version": "6.0.1",
I/flutter (21900):   "sdk": 23
I/flutter (21900): }

# samsung S8
I/flutter (22297): {
I/flutter (22297):   "board": "msm8998",
I/flutter (22297):   "bootloader": "G950U1UEU5DSC1",
I/flutter (22297):   "brand": "samsung",
I/flutter (22297):   "device": "dreamqlteue",
I/flutter (22297):   "hardware": "qcom",
I/flutter (22297):   "host": "SWDH7919",
I/flutter (22297):   "isPhysicalDevice": true,
I/flutter (22297):   "manufacturer": "samsung",
I/flutter (22297):   "model": "SM-G950U1",
I/flutter (22297):   "product": "dreamqlteue",
I/flutter (22297):   "tags": "release-keys",
I/flutter (22297):   "type": "user",
I/flutter (22297):   "version": "9",
I/flutter (22297):   "sdk": 28
I/flutter (22297): }

# samsung S10+
I/flutter (10037): {
I/flutter (10037):   "board": "universal9820",
I/flutter (10037):   "bootloader": "G975FXXS3ASJG",
I/flutter (10037):   "brand": "samsung",
I/flutter (10037):   "device": "beyond2",
I/flutter (10037):   "hardware": "exynos9820",
I/flutter (10037):   "host": "SWDG4719",
I/flutter (10037):   "isPhysicalDevice": true,
I/flutter (10037):   "manufacturer": "samsung",
I/flutter (10037):   "model": "SM-G975F",
I/flutter (10037):   "product": "beyond2ltexx",
I/flutter (10037):   "tags": "release-keys",
I/flutter (10037):   "type": "user",
I/flutter (10037):   "version": "9",
I/flutter (10037):   "sdk": 28
I/flutter (10037): }

Not working on these devices

# pixel XL
I/flutter ( 1500): {
I/flutter ( 1500):   "board": "marlin",
I/flutter ( 1500):   "bootloader": "8996-012001-1904111134",
I/flutter ( 1500):   "brand": "google",
I/flutter ( 1500):   "device": "marlin",
I/flutter ( 1500):   "hardware": "marlin",
I/flutter ( 1500):   "host": "abfarm826",
I/flutter ( 1500):   "isPhysicalDevice": true,
I/flutter ( 1500):   "manufacturer": "Google",
I/flutter ( 1500):   "model": "Pixel XL",
I/flutter ( 1500):   "product": "marlin",
I/flutter ( 1500):   "tags": "release-keys",
I/flutter ( 1500):   "type": "user",
I/flutter ( 1500):   "version": "9",
I/flutter ( 1500):   "sdk": 28
I/flutter ( 1500): }

# oneplus 6
I/flutter (15401): {
I/flutter (15401):   "board": "sdm845",
I/flutter (15401):   "bootloader": "unknown",
I/flutter (15401):   "brand": "OnePlus",
I/flutter (15401):   "device": "OnePlus6",
I/flutter (15401):   "hardware": "qcom",
I/flutter (15401):   "host": "rd-build-73",
I/flutter (15401):   "isPhysicalDevice": true,
I/flutter (15401):   "manufacturer": "OnePlus",
I/flutter (15401):   "model": "ONEPLUS A6000",
I/flutter (15401):   "product": "OnePlus6",
I/flutter (15401):   "tags": "release-keys",
I/flutter (15401):   "type": "user",
I/flutter (15401):   "version": "10",
I/flutter (15401):   "sdk": 29
I/flutter (15401): }

flutter doctor -v

[✓] Flutter (Channel stable, v1.9.1+hotfix.5, on Mac OS X 10.14.6 18G95, locale en-US)
    • Flutter version 1.9.1+hotfix.5 at /Users/corey/flutter
    • Framework revision 1aedbb1835 (6 weeks ago), 2019-10-17 08:37:27 -0700
    • Engine revision b863200c37
    • Dart version 2.5.0

[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
    • Android SDK at /Users/corey/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-28, build-tools 28.0.3
    • ANDROID_HOME = /Users/corey/Library/Android/sdk
    • ANDROID_SDK_ROOT = /Users/corey/Library/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b49-5587405)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.2.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.2.1, Build version 11B500
    • CocoaPods version 1.7.5

[✓] Android Studio (version 3.5)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 40.2.2
    • Dart plugin version 191.8593
    • Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b49-5587405)

[✓] VS Code (version 1.40.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.6.0

[✓] Connected device (4 available)
    • ONEPLUS A6000 • 4241cff9           • android-arm64 • Android 10 (API 29)
    • SM J700H      • 52033d7fe8448323   • android-arm   • Android 6.0.1 (API 23)
    • SM G950U1     • 988939444c58415246 • android-arm64 • Android 9 (API 28)
    • Pixel XL      • HT76E0201647       • android-arm64 • Android 9 (API 28)

• No issues found!

Posted on Stack Overflow as well in case I'm doing something wrong and this isn't actually a bug with firebase ML vision

The text was updated successfully, but these errors were encountered:

CoreyCole · 2019-12-02T18:41:58Z

We fixed the issue by changing the ResolutionPreset on the CameraController to high:

final newCameraController = CameraController(
    cameras.first,
    ResolutionPreset.high,
    enableAudio: false,
);

We are only using the camera preview for scanning a QR code, so high resolution is overkill for our use case. On lower-end devices it causes dropped frames to set the camera controller to high. Would love to be able to use this plugin on lower resolution presets.

eliasjtg · 2020-01-02T16:48:44Z

Same issue, high resolution is overkill, medium crash

rmtmckenzie · 2020-01-09T14:40:40Z

I'm seeing the same thing with a Nexus 5x, on medium it doesn't find anything.

CoreyCole · 2020-01-09T18:02:16Z

For anyone else looking only to scan QR codes, we decided to use qrcode. It uses native packages AVCaptureSession in iOS and zxing in Android.

https://pub.dev/packages/qrcode

Running the firebase_ml_vision for QR codes was overkill, even more so with the resolution pinned at high.

claireliu14 · 2020-02-03T07:25:19Z

I met the issue when I run app on pixel 3a with camera streaming and ml_vision.
If ResolutionPresent is setting lower than high, ml_vision will detect nothing.

But this problem not appear when I debug on Samsung s8 and iPhone 6s.
They can only work when I set ResolutionPresent to medium.
CameraPreview will lag when ResolutionPresent set high.

Any solution ?

sgehrman · 2020-02-15T02:29:04Z

Same issue, the example app doesn't even work on Android.

greenlihui · 2020-02-21T05:58:03Z

Similar thing also happened here, I am writing a real time face detection app using face detector, and I was setting ResolutionPreset to medium, which results no faces detected at all, it took me days to find out the reason. My device is Samsung S9 running Android X(api 29), face detection working fine if preset is not set to medium.

topnax · 2020-03-19T17:19:43Z

I can relate to this. Text OCR works on ResolutionPreset.HIGH, but on slower devices it might stutter a bit. On Samsung S8 it stutters hell a lot and is completely unusable. Switching to ResolutionPreset.MEDIUM fixes the Samsung S8 issue, but completely disables other devices as they no longer recognize a single character from the image.

benjastudio · 2020-04-16T14:22:23Z

Same Problem here. The medium resolution was not working at all.

I looked closer at what happened, and I found that my phone (redmi 4X) added a padding to each row of the planes when the medium resolution was selected (but not with high or low resolution).

In my case the image format is a YUV 4:2:0, meaning the image is decomposed in 3 planes (Y, U, V), with 8 bits per pixel for Y, and 4 for U and V.

Each plane is a matrix of ~pixel stored in a 1D-array. They come with a bytesPerRow value that represents the length of a row. This may be or not the same size than the image width depending if some padding is added at the end of each row or not. On my case, bytesPerRow was 768and my image width 720 (so a padding of 48 bytes has been added to each row).

As bytesPerRow is given to the FirebaseVisionImagePlaneMetadata object I suppose that the row stride should be handled correctly by the library. But it looks that something is not handled right?

1 - I can't help and dig more into this than I actually did.

2 - I wrote 2 workarounds (A and B) that keep the data consistency:

A: Remove manually the padding in the plane (for each row) and set the bytesPerRow to the image width. Drawback: It's a bit painful and I suppose it may slow down a bit the process.
B: Change the image width to include the padding. Drawback: your image has a "black border" on the right of the image of the size of the padding. It may not be a problem in some cases.

// For each plane
FirebaseVisionImagePlaneMetadata(
        bytesPerRow: plane.bytesPerRow,
        width: plane.bytesPerRow,
        height: image.height,
      )

...

  int newWidth = planes[0].bytesPerRow;
  // Build image metadata
  var metadata = FirebaseVisionImageMetadata(
    rawFormat: image.format.raw,
    size: Size(newWidth.toDouble(), image.height.toDouble()),
    rotation: ImageRotation.rotation90,
    planeData: planesMetadata,
  );
...

Both are working for my own but they still are workaround.
I hope this will help some of you.

SubashManian · 2020-05-11T15:38:46Z

Bar Code scanner won't run iPhone 8 device while set it max resolution. if i set resolution to medium at the time it work's fine but can't able to scan the lower quality bar code in printed copies. while set to max resolution iPhone 6, 6s, 7, 7 plus, 8, 8 plus memory consumption is more than gb and it leads to crash. Attached Memory usage screen shot.
please advice

decolon · 2020-06-05T20:38:58Z

I have the same problem on my Pixel 3a. Only works with max resolution, and in max resolution the memory builds until the app crashes.

AHuminskyi · 2020-07-14T12:54:29Z

We fixed the issue by changing the ResolutionPreset on the CameraController to high:
final newCameraController = CameraController(
    cameras.first,
    ResolutionPreset.high,
    enableAudio: false,
);
We are only using the camera preview for scanning a QR code, so high resolution is overkill for our use case. On lower-end devices it causes dropped frames to set the camera controller to high. Would love to be able to use this plugin on lower resolution presets.

ResolutionPreset.high or even ultra didn't help me on OnePlus 6t :(

IliaKhuzhakhmetovRoonyx · 2020-10-12T10:40:00Z

whats new?

svenjacobs · 2020-12-03T12:22:20Z

I'm having the same problem on a Google Pixel (1) and Pixel 4 XL. Only with ResolutionPreset.high are barcodes detected reliably.

TheArchitect123 · 2021-01-15T06:46:21Z

whats new?

nothing, by the looks of it. This issue has been opened for over a year now, so I'm assuming that nobody's looking at this. If I were you I would be looking at using another library

CoreyCole added the type: bug Something isn't working label Nov 29, 2019

CoreyCole mentioned this issue Dec 2, 2019

FlutterFire ML Vision Barcodes only works on ResolutionPreset.high CameraController flutter/flutter#45825

Closed

CoreyCole changed the title ~~[firebase_ml_vision] Only works on my Samsung Devices~~ [firebase_ml_vision] Only works on ResolutionPreset.high CameraController Dec 2, 2019

iapicca added the type: enhancement New feature or request label Dec 4, 2019

greenlihui mentioned this issue Feb 21, 2020

Face detect not work with medium resolution preset flutter/flutter#38683

Closed

Ehesp added impact: crowd Affects many people, though not necessarily a specific customer with an assigned label. (P2) plugin: ml_vision labels Apr 20, 2020

PolinaGo added the ml_on_device label Jul 24, 2020

This was referenced Mar 28, 2021

🐛 [firebase_ml_vision] Label detector only detecting "metal" and "pattern" #5501

Closed

docs: Using high preset for camera preview example on android. #5507

Closed

docs: Upgrading to camera plugin 0.8.1 #5520

Merged

Salakar closed this as completed Apr 1, 2021

firebase locked and limited conversation to collaborators Apr 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

This issue was moved to a discussion.

[firebase_ml_vision] Only works on ResolutionPreset.high CameraController #1518

[firebase_ml_vision] Only works on ResolutionPreset.high CameraController #1518

CoreyCole commented Nov 29, 2019 •

edited

CoreyCole commented Dec 2, 2019 •

edited

eliasjtg commented Jan 2, 2020

rmtmckenzie commented Jan 9, 2020

CoreyCole commented Jan 9, 2020

claireliu14 commented Feb 3, 2020 •

edited

sgehrman commented Feb 15, 2020

greenlihui commented Feb 21, 2020

topnax commented Mar 19, 2020

benjastudio commented Apr 16, 2020 •

edited

SubashManian commented May 11, 2020

decolon commented Jun 5, 2020

AHuminskyi commented Jul 14, 2020 •

edited

IliaKhuzhakhmetovRoonyx commented Oct 12, 2020

svenjacobs commented Dec 3, 2020

TheArchitect123 commented Jan 15, 2021

This issue was moved to a discussion.

This issue was moved to a discussion.

[firebase_ml_vision] Only works on ResolutionPreset.high CameraController #1518

[firebase_ml_vision] Only works on ResolutionPreset.high CameraController #1518

Comments

CoreyCole commented Nov 29, 2019 • edited

CoreyCole commented Dec 2, 2019 • edited

eliasjtg commented Jan 2, 2020

rmtmckenzie commented Jan 9, 2020

CoreyCole commented Jan 9, 2020

claireliu14 commented Feb 3, 2020 • edited

sgehrman commented Feb 15, 2020

greenlihui commented Feb 21, 2020

topnax commented Mar 19, 2020

benjastudio commented Apr 16, 2020 • edited

SubashManian commented May 11, 2020

decolon commented Jun 5, 2020

AHuminskyi commented Jul 14, 2020 • edited

IliaKhuzhakhmetovRoonyx commented Oct 12, 2020

svenjacobs commented Dec 3, 2020

TheArchitect123 commented Jan 15, 2021

This issue was moved to a discussion.

CoreyCole commented Nov 29, 2019 •

edited

CoreyCole commented Dec 2, 2019 •

edited

claireliu14 commented Feb 3, 2020 •

edited

benjastudio commented Apr 16, 2020 •

edited

AHuminskyi commented Jul 14, 2020 •

edited