Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug report] NullPointerException when reading QR code from specific image #569

Closed
AlexV525 opened this issue Sep 28, 2022 · 15 comments
Closed
Assignees

Comments

@AlexV525
Copy link

AlexV525 commented Sep 28, 2022

Describe the bug
When I was trying to read a specific image file that contains a valid QR code, the scanner throws an NPE or is unable to read the data.

D/VisionProcessorBase: Failed to process. Error: Internal error has occurred when executing ML Kit tasks
W/System.err: com.google.mlkit.common.MlKitException: Internal error has occurred when executing ML Kit tasks
W/System.err:     at com.google.mlkit.common.sdkinternal.ModelResource.zza(com.google.mlkit:common@@18.5.0:9)
W/System.err:     at com.google.mlkit.common.sdkinternal.zzn.run(Unknown Source:10)
W/System.err:     at com.google.mlkit.common.sdkinternal.zzt.run(com.google.mlkit:common@@18.5.0:2)
W/System.err:     at com.google.mlkit.common.sdkinternal.MlKitThreadPool.zze(com.google.mlkit:common@@18.5.0:4)
W/System.err:     at com.google.mlkit.common.sdkinternal.MlKitThreadPool.zzc(com.google.mlkit:common@@18.5.0:1)
W/System.err:     at com.google.mlkit.common.sdkinternal.zzk.run(Unknown Source:2)
W/System.err:     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
W/System.err:     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
W/System.err:     at com.google.mlkit.common.sdkinternal.MlKitThreadPool.zzd(com.google.mlkit:common@@18.5.0:2)
W/System.err:     at com.google.mlkit.common.sdkinternal.zzi.run(Unknown Source:2)
W/System.err:     at java.lang.Thread.run(Thread.java:923)
W/System.err: Caused by: java.lang.NullPointerException: Attempt to get length of null array
W/System.err:     at com.google.android.gms.internal.mlkit_vision_barcode_bundled.zzec.zzJ(com.google.mlkit:barcode-scanning@@17.0.2:1)
W/System.err:     at com.google.photos.vision.barhopper.BarhopperProto$BarhopperResponse.zzb(com.google.mlkit:barcode-scanning@@17.0.2:1)
W/System.err:     at com.google.android.libraries.barhopper.BarhopperV3.toProto(com.google.mlkit:barcode-scanning@@17.0.2:1)
W/System.err:     at com.google.android.libraries.barhopper.BarhopperV3.recognize(com.google.mlkit:barcode-scanning@@17.0.2:11)
W/System.err:     at com.google.mlkit.vision.barcode.bundled.internal.zza.zzb(com.google.mlkit:barcode-scanning@@17.0.2:5)
W/System.err:     at com.google.android.gms.internal.mlkit_vision_barcode_bundled.zzbk.zza(com.google.mlkit:barcode-scanning@@17.0.2:3)
W/System.err:     at com.google.android.gms.internal.mlkit_vision_barcode_bundled.zzb.onTransact(com.google.mlkit:barcode-scanning@@17.0.2:3)
W/System.err:     at android.os.Binder.transact(Binder.java:1054)
W/System.err:     at com.google.android.gms.internal.mlkit_vision_barcode.zza.zzb(com.google.android.gms:play-services-mlkit-barcode-scanning@@18.0.0:2)
W/System.err:     at com.google.android.gms.internal.mlkit_vision_barcode.zzox.zzd(com.google.android.gms:play-services-mlkit-barcode-scanning@@18.0.0:4)
W/System.err:     at com.google.mlkit.vision.barcode.internal.zzl.zza(com.google.android.gms:play-services-mlkit-barcode-scanning@@18.0.0:14)
W/System.err:     at com.google.mlkit.vision.barcode.internal.zzi.zzc(com.google.android.gms:play-services-mlkit-barcode-scanning@@18.0.0:3)
W/System.err:     at com.google.mlkit.vision.barcode.internal.zzi.run(com.google.android.gms:play-services-mlkit-barcode-scanning@@18.0.0:1)
W/System.err:     at com.google.mlkit.vision.common.internal.MobileVisionBase.zza(com.google.mlkit:vision-common@@17.2.1:2)
W/System.err:     at com.google.mlkit.vision.common.internal.zza.call(Unknown Source:4)
W/System.err:     at com.google.mlkit.common.sdkinternal.ModelResource.zza(com.google.mlkit:common@@18.5.0:8)
W/System.err: 	... 10 more
E/BarcodeProcessor: Barcode detection failed com.google.mlkit.common.MlKitException: Internal error has occurred when executing ML Kit tasks

To Reproduce

The image contains some personal private info, if it's applicable, please leave an email address then I can send the original image.

Example Steps to reproduce the behavior in the sample app:

  1. Run the "ML Kit Vision Quickstart" example.
  2. Click on 'Run the ML kit quickstart written in Kotlin'
  3. Click on 'StillImageActivity'
  4. Change "Object Detection" to "Barcode Scanning"
  5. Choose the image from the album.

SDK Info:

  • com.google.mlkit:barcode-scanning:17.0.2
  • GoogleMLKit/BarcodeScanning ~3.2.0

Smartphone:

  • Device/Simulator: All devices (including Android and iOS)
@zhouyiself
Copy link
Collaborator

Thanks for the report! Can you share your image with zhouyiself@gmail.com?

@AlexV525
Copy link
Author

Thanks for the report! Can you share your image with zhouyiself@gmail.com?

@zhouyiself Sure. The image has been sent.

@zeroDivider
Copy link

zeroDivider commented Oct 6, 2022

This happened to me too, will it be considered a d fixed soon?
I am using 18.1.0. It worked well when we were using 16.2.1.

@SilverDestiny
Copy link
Collaborator

Contacted @AlexV525 by email. Please share the image with me to reproduce this issue. Thanks!

@zhouyiself
Copy link
Collaborator

@AlexV525 Hi, I didn't see your image in the email list.

@AlexV525
Copy link
Author

AlexV525 commented Oct 7, 2022

@AlexV525 Hi, I didn't see your image in the email list.

Can you check again?
image

@SilverDestiny
Copy link
Collaborator

The root cause is the input Bitmap config is not ARGB_8888, and our engine only supports config ARGB_8888 for Bitmap underneath: https://developer.android.com/reference/android/graphics/Bitmap.Config

It works well for streaming mode, i.e. LivePreviewActivity and CameraXLivePreviewActivity

We'll consider converting Bitmap config to ARGB_8888 if not internally to fix this issue in later releases. Thanks for reporting this bug!

@SilverDestiny
Copy link
Collaborator

BTW, the original input Bitmap config is RGBA_F16

@googlesamples googlesamples deleted a comment Oct 10, 2022
@AlexV525
Copy link
Author

AlexV525 commented Oct 11, 2022

We'll consider converting Bitmap config to ARGB_8888 if not internally to fix this issue in later releases. Thanks for reporting this bug!

BitmapFactory.Options.outConfig (link) only supported from API 26, so it'll be good if we can convert the Bitmap if it's not ARGB_8888 in the ML Kit internally.

@SilverDestiny
Copy link
Collaborator

https://developer.android.com/reference/android/graphics/Bitmap#getConfig() could be used to get the config from bitmap?

@AlexV525
Copy link
Author

https://developer.android.com/reference/android/graphics/Bitmap#getConfig() could be used to get the config from bitmap?

Yes, and it requires a memory allocation to read the actual data in IIRC.

@AlexV525
Copy link
Author

AlexV525 commented Oct 11, 2022

Workaround with an actual allocation:

val mlImage: InputImage
val shouldConvertBitmap: Boolean
var decodedBitmap: Bitmap? = null
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.O) {
    // Decode the bitmap without memory allocation.
    val options = BitmapFactory.Options().apply { inJustDecodeBounds = true }
    BitmapFactory.decodeFile(path, options)
    shouldConvertBitmap = options.outConfig != Bitmap.Config.ARGB_8888
} else {
    val bitmap = BitmapFactory.decodeFile(path)
    shouldConvertBitmap = bitmap.config != Bitmap.Config.ARGB_8888
    // Cache the decoded bitmap.
    decodedBitmap = bitmap
}
mlImage = if (shouldConvertBitmap) {
    val bitmap = decodedBitmap ?: BitmapFactory.decodeFile(path)
    val copiedBitmap = bitmap.copy(Bitmap.Config.ARGB_8888, false)
    // Recycle the original bitmap.
    bitmap.recycle()
    InputImage.fromBitmap(copiedBitmap, 0)
} else {
    InputImage.fromFilePath(this, Uri.fromFile(File(path)))
}
// Recycle the decoded bitmap if it exists.
decodedBitmap?.recycle()

@AlexV525
Copy link
Author

AlexV525 commented Nov 2, 2022

@miworking3 @SilverDestiny This also happened on iOS, so iOS should provide the same ability to convert if images are not ARGB_8888.

@AlexV525
Copy link
Author

AlexV525 commented Nov 2, 2022

iOS workaround (I don't use unwrapping here, feel free to modify if it's worth in your code):

var image: UIImage? = UIImage.init(contentsOfFile: path)
if (image == nil) {
    // Throw something.
}
let fileExtension = URL(fileURLWithPath: path).pathExtension.lowercased()
// Convert all PNG to JPEG to avoid unsupported formats from ML Kit.
if (fileExtension == "png") {
    let imageData = image!.jpegData(compressionQuality: 1)
    image = UIImage(data: imageData!)
    if (image == nil) {
        // Throw something.
    }
}
let visionImage = VisionImage(image: image!)
visionImage.orientation = image!.imageOrientation

@SilverDestiny
Copy link
Collaborator

The latest ML Kit release fixed this issue: https://developers.google.com/ml-kit/release-notes#december_6_2022

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants