TextRecognizer works ok on Android but the result letters are all corrupted on iOS

[kodol73]

Hi,

I got recommended to use google_mlkit_text_recognition instead of google_ml_kit as followes
#207

However, it still works ok on Android but not on iOS using google_mlkit_text_recognition as using google_ml_kit

recognizedText.text look ok on Android but the letters in it are all corrupted on iOS
For now, I need only English TextRecognizer

import 'package:google_mlkit_text_recognition/google_mlkit_text_recognition.dart';

textRecognizer = TextRecognizer(script: TextRecognitionScript.latin);
final File? imageFile = await getImage(ImageSource.gallery);

if(imageFile != null) {
  final inputImage = InputImage.fromFile(imageFile);
  final RecognizedText recognizedText = await textRecognizer.processImage(inputImage);

  String text = recognizedText.text;
  print('${this.className} extractText : ' + text);

}

google_mlkit_text_recognition 0.0.1

My development environment is as followes
iOS 14.7.1
Xcode 13.3

Please help me to solve this issue
Thanks in advance

[fbernaly]

@kodol73 : could you share an image of the text you are trying to recognize? I have been trying text recognizer and it works for me.
Also, I see in your code that you are using and image from the gallery, could you try the example app using the camera feed?

[fbernaly]

Here you have some screenshoot of the example app in iOS and it is working.

[pdhiepCMCJapan]

i have a same problem, it detect wrong text when run on IOS flutter

[kodol73]

It seems that some images of text are recognized correctly, but some other images of text are corrupted
I just tried with example app, but has the same result

The result of recognizing below image is corrupted
Please check with the below image

[fbernaly]

@kodol73 : I tried your image with the example app and so far it does not look corrupted to me. It looks pretty much good to me.

[kodol73]

Can you please let me know your phone model, iOS version and Xcode version?

[fbernaly]

I tried with both an iPhone 11 and iPhone 6S, both with iOS 15 and I am using Xcode 13.3

[kodol73]

I tried with both an iPhone 11 and iPhone 6S, both with iOS 15 and I am using Xcode 13.3

it's weird, I just upgraded iOS to 15.4.1, but the same result
It generated corrupted text like the following with the image

T
T
ooo
5

Just in case, I test with google_ml_kit 0.8.0 because example app uses google_ml_kit

Please help me to solve this issue

[fbernaly]

@kodol73: I don't know how to help you more. I have tested in many iOS devices and in all cases text recognition in the sample app is working ok.

Devices I have tested:

• iPhone 6s
• iPhone 11
• iPad pro
• iPad air
• iPhone SE

I hope you can find out why your images are not working. This plugin is just wrapping the native APIs using FlutterMethodChannel as explained here. In other words, it is taking the image output from the Camera plugin or image_picker plugin, and passing it to the FlutterMethodChannel for processing in the native layer using the APIs as explained here, then passing the response back to Flutter. We are not doing any extra processing to the image.

To discard that the issue is in the Flutter or the native layer I would run a native iOS app using the pod for TextRecognition and make sure that it is working or not. If it is not working and you still get the same issue then I would say that the issue is in the ML native layer. If native app works then I would move to debug the Flutter layer and see why the images are not being pass correctly... I will move every layer of the plugin until I find which one is causing the issue. Once you find it come back and report it or even better: fix it and send us a PR with your contribution.

Here you have a native iOS sample app: https://github.com/googlesamples/mlkit/tree/master/ios/quickstarts/vision

So far I can not help you more because I cannot reproduce the issue. Also, I am supporting this plugin for fun and have other duties.

BTW, if you only need text recognition you should start using google_mlkit_text_recognition instead. More details why here:

• https://github.com/bharat-biradar/Google-Ml-Kit-plugin/tree/master/packages/google_ml_kit
• https://github.com/bharat-biradar/Google-Ml-Kit-plugin/tree/master/packages/google_mlkit_text_recognition

[codyfilatov]

I had this same issue, but resolved it on iOS.

Still testing android but I am passing the File from camera plugin cameraController.takePicture() as the InputImage.fromFile(imageFile) and looking at text it had almost no relevant data, and any data it did have was like:
A
ooD
TM

After removing the google_ml_kit plugin, and replacing it with only the google_mlkit_text_recognition: ^0.2.0 it works fine. No idea why to be honest, but specifying just text recognition instead of the umbrella plugin resolved. Would like to hear some details on this if there are any.

[fbernaly]

Thanks @codyfilatov for the suggestion.

Do you still have issues with Android? both platforms are working okay?

@kodol73 : give it a try

[codyfilatov]

Both platforms are working okay now.
Although I did not have this issue with Android to begin with.

The only issue I am having now on iOS is the orientation of my image when processing, it thinks it is in landscape when it is in portrait, so the bounding boxes are messed up, does not happen on Android. My image does not have any exif orientation data. I would be nice to have if no orientation data exists to predict based on the height/width, looks like it defaults to Landscape if no data exists. But that's another issue I can figure out.

Hope OP resolves issue with the dependency change.
Thanks

[fbernaly]

@codyfilatov: once you figure the orientation out, share here or create another issue if necessary. I have seen issues with different orientations as well.

[kodol73]

I just tested the newly released version of google_mlkit_text_recognition, 0.2.0, nothing else changed
The issue doesn't appear any more

Thanks

[codyfilatov]

@codyfilatov: once you figure the orientation out, share here or create another issue if necessary. I have seen issues with different orientations as well.

Sharing here just as a note.
The orientation issue was caused by no EXIF orientation data on the image in iOS taken from the camera plugin takePicture function. Since the image file had no EXIF orientation data I assume the text regognizer (and barcode scanner) was returning odd (not incorrect) values.

The values did not translate as expected as when I set a Container DecorationImage as the same file the image is displayed correctly as taken, but the values were printed mirrored and counter clockwise consistently.

After using the image package to set the EXIF orientation data to 1 and rewriting the bytes to a new file, then passing that file to MLkit the boundingBox values were as expected.

[fbernaly]

@codyfilatov: is there any code you can share of how you did it?

[codyfilatov]

XFile imageCaptureXFile = await this.cameraController!.takePicture();
File imageFile = File(imageCaptureXFile.path);
this.imageFile = imageFile;
img.Image? image = img.decodeImage(await imageFile.readAsBytes());
if(image == null){
    Util.showToast('Image was not found.');
    this.isConfirming = false;
    this.imageFile = null;
    setState(() {});
    return;
}
if(!image.exif.hasOrientation){
    print('Baking orientation into image...');
    image.exif.orientation = 1;
    File orientatedImageFile = File((await getTemporaryDirectory()).path + Uuid().v4() + '.jpg');
    await orientatedImageFile.create();
    await orientatedImageFile.writeAsBytes(img.encodeJpg(image));
    await imageFile.delete();
    imageFile = orientatedImageFile;
    this.imageFile = orientatedImageFile;

    image = img.decodeImage(await orientatedImageFile.readAsBytes());
    if(image == null){
        setState(() {});
        return;
    }
}

This is what I use for my use case. Keep in mind I only want portrait.
I use the package:image/image.dart . Ignore the bad code, I'll clean it up later.

TextRecognizer textDetector = TextRecognizer(script: TextRecognitionScript.latin);
RecognizedText recognisedText = await textDetector.processImage(InputImage.fromFile(imageFile));

[fbernaly]

@codyfilatov : are using live camera feed? taking picture with the camera? or from gallery?

[fbernaly]

Closing this issue.

Fix: google_mlkit_text_recognition instead of google_ml_kit.

More details here: https://github.com/bharat-biradar/Google-Ml-Kit-plugin/tree/master/packages/google_mlkit_text_recognition

Moving InputImage and InputImageRotation issues to: #240