RFE: Support Japanese and other non-Latin scripts in the built-in TextRecognizer

[jsfan3]

Motivation

The new built-in on-device vision APIs introduced in #5467 look very useful, and I would like to use com.codename1.ai.vision.TextRecognizer for OCR of Japanese text.

At the moment, however, the portable API doesn't seem to provide a way to select the recognition language or writing system.

In particular, the current Android implementation uses ML Kit's Latin TextRecognizerOptions.DEFAULT_OPTIONS, so the built-in TextRecognizer cannot be used for Japanese OCR on Android.

This appears to be mainly a missing portable API/configuration option rather than a limitation of the underlying native frameworks. ML Kit Text Recognition v2 provides dedicated recognizers for Latin, Chinese, Devanagari, Japanese, and Korean scripts. Apple Vision also provides language configuration for text recognition.

Request

Could Codename One expose a portable way to configure the language or writing system used by TextRecognizer?

Thank you!

[shai-almog]

Sure, we'll take a look at it but why didn't you file an RFE in the issue tracker?

[shai-almog]

This is reproducible though, right? The phrasing might be off there.

[jsfan3]

This is the link to reproduce: https://github.com/codenameone/CodenameOne/issues/new/choose
Alternatively, click on “Issues” and then on “New issue”.

[shai-almog]

I meant the original problem. If it's reproducible then the RFE should be the right direction.

[jsfan3]

Yes, the original OCR problem is reproducible. I tested it on Android with a fresh Codename One project created with CN1 Initializr.

I only used Android as a concrete example because it is the relevant platform for my use case. However, I believe the RFE is applicable to the other build targets as well.

1. Create the project

Create a new Java application with CN1 Initializr using:

Package:    com.example.ocrrfe
Main class: OcrRfeTest

Save these two deterministic test images:

• ocr_english.png

• ocr_japanese_italian.png

into:

common/src/main/resources/

2. Add this portable test application

Replace the generated main class with:

package com.example.ocrrfe;

import com.codename1.ai.vision.TextRecognizer;
import com.codename1.ai.vision.VisionImage;
import com.codename1.io.Log;
import com.codename1.io.Util;
import com.codename1.system.Lifecycle;
import com.codename1.system.NativeLookup;
import com.codename1.ui.Button;
import com.codename1.ui.Display;
import com.codename1.ui.Form;
import com.codename1.ui.Label;
import com.codename1.ui.TextArea;
import com.codename1.ui.layouts.BoxLayout;

import java.io.IOException;
import java.io.InputStream;

public class OcrRfeTest extends Lifecycle {

    private byte[] englishImage;
    private byte[] japaneseItalianImage;
    private TextArea output;

    @Override
    public void runApp() {
        try {
            englishImage = load("/ocr_english.png");
            japaneseItalianImage = load("/ocr_japanese_italian.png");
        } catch (IOException ex) {
            Log.e(ex);
            return;
        }

        Form form = new Form("OCR RFE", BoxLayout.y());

        form.add(new Label("Same images, two recognizers"));

        Button builtInEnglish =
                new Button("1. Built-in TextRecognizer: English");

        Button builtInMixed =
                new Button("2. Built-in TextRecognizer: Japanese + Italian");

        Button japaneseMixed =
                new Button("3. ML Kit Japanese recognizer: Japanese + Italian");

        output = new TextArea("Ready");
        output.setEditable(false);
        output.setRows(16);

        builtInEnglish.addActionListener(
                e -> runBuiltIn("BUILT-IN / ENGLISH", englishImage));

        builtInMixed.addActionListener(
                e -> runBuiltIn("BUILT-IN / JAPANESE + ITALIAN",
                        japaneseItalianImage));

        japaneseMixed.addActionListener(
                e -> runJapaneseNative(japaneseItalianImage));

        form.add(builtInEnglish);
        form.add(builtInMixed);
        form.add(japaneseMixed);
        form.add(output);

        form.show();
    }

    private void runBuiltIn(String label, byte[] imageBytes) {
        final TextRecognizer recognizer = new TextRecognizer();

        if (!recognizer.isSupported()) {
            output.setText(label + "\nUNSUPPORTED");
            recognizer.close();
            return;
        }

        output.setText(label + "\nRunning...");

        recognizer.process(VisionImage.encoded(imageBytes))
                .ready(result -> {
                    String text = result.getText();
                    String message = label + "\n\n" + text;
                    Log.p(message);
                    output.setText(message);
                    recognizer.close();
                })
                .except(error -> {
                    Log.e(error);
                    output.setText(label + "\nERROR: " + error);
                    recognizer.close();
                });
    }

    private void runJapaneseNative(byte[] imageBytes) {
        final JapaneseOcrNative nativeOcr =
                NativeLookup.create(JapaneseOcrNative.class);

        if (nativeOcr == null || !nativeOcr.isSupported()) {
            output.setText("JAPANESE ML KIT\nUNSUPPORTED");
            return;
        }

        output.setText("JAPANESE ML KIT\nRunning...");

        Display.getInstance().scheduleBackgroundTask(() -> {
            final String result = nativeOcr.recognize(imageBytes);
            final String message =
                    "JAPANESE ML KIT / JAPANESE + ITALIAN\n\n" + result;

            Log.p(message);

            Display.getInstance().callSerially(
                    () -> output.setText(message));
        });
    }

    private byte[] load(String resource) throws IOException {
        InputStream in =
                Display.getInstance().getResourceAsStream(
                        getClass(), resource);

        if (in == null) {
            throw new IOException("Resource not found: " + resource);
        }

        try {
            return Util.readInputStream(in);
        } finally {
            Util.cleanup(in);
        }
    }
}

3. Add a minimal NativeInterface for the comparison

Create:

common/src/main/java/com/example/ocrrfe/JapaneseOcrNative.java

with:

package com.example.ocrrfe;

import com.codename1.system.NativeInterface;

public interface JapaneseOcrNative extends NativeInterface {
    String recognize(byte[] encodedImage);
}

From the root of the Maven project run:

mvn -Dcodename1.platform=android cn1:generate-native-interfaces

This generates the Android implementation stub at:

android/src/main/java/com/example/ocrrfe/JapaneseOcrNativeImpl.java

4. Add the Japanese ML Kit dependency

Add these build hints:

android.min_sdk_version=23
android.gradleDep=implementation 'com.google.mlkit:text-recognition-japanese:16.0.1'

Google's current Text Recognition v2 Android documentation lists API 23+ and the bundled Japanese dependency above.

5. Replace only the generated Android implementation

Replace:

android/src/main/java/com/example/ocrrfe/JapaneseOcrNativeImpl.java

with:

package com.example.ocrrfe;

import android.graphics.Bitmap;
import android.graphics.BitmapFactory;

import com.google.android.gms.tasks.Tasks;
import com.google.mlkit.vision.common.InputImage;
import com.google.mlkit.vision.text.Text;
import com.google.mlkit.vision.text.TextRecognition;
import com.google.mlkit.vision.text.TextRecognizer;
import com.google.mlkit.vision.text.japanese.JapaneseTextRecognizerOptions;

public class JapaneseOcrNativeImpl {

    public String recognize(byte[] encodedImage) {
        Bitmap bitmap = BitmapFactory.decodeByteArray(
                encodedImage, 0, encodedImage.length);

        if (bitmap == null) {
            return "ERROR: could not decode image";
        }

        InputImage image = InputImage.fromBitmap(bitmap, 0);

        TextRecognizer recognizer = TextRecognition.getClient(
                new JapaneseTextRecognizerOptions.Builder().build());

        try {
            Text result = Tasks.await(recognizer.process(image));
            return result.getText();
        } catch (Exception ex) {
            return "ERROR: " + ex;
        } finally {
            recognizer.close();
        }
    }

    public boolean isSupported() {
        return true;
    }
}

The relevant part is only:

TextRecognition.getClient(
        new JapaneseTextRecognizerOptions.Builder().build());

6. Build and run on Android

The result is a 50 MiB APK file.

7. Testing

Test 1: built-in TextRecognizer, English image

The built-in recognizer reads the English image correctly.

Screenshot:

So the built-in OCR itself is working.

Test 2: built-in TextRecognizer, Japanese + Italian image

With the same built-in API, the English/Italian parts are recognized, while the Japanese text is not correctly recognized.

Screenshot:

This is consistent with the current Android implementation of Codename One, which imports:

com.google.mlkit.vision.text.latin.TextRecognizerOptions

and creates the recognizer with:

private final TextRecognizer client = TextRecognition.getClient(
        TextRecognizerOptions.DEFAULT_OPTIONS);

TextRecognizerOptions is the Latin-only ML Kit recognizer configuration.

Current CN1 source:
AndroidTextRecognitionAdapter.java

Test 3: same Japanese + Italian image, Japanese ML Kit options

Using the same image bytes but creating the ML Kit client with:

new JapaneseTextRecognizerOptions.Builder().build()

the Japanese text is recognized, while the Latin/Italian text continues to be recognized too.

The result is substantially:

Google documents JapaneseTextRecognizerOptions as recognizing Latin and Japanese.

Relevant Google documentation:

• ML Kit Text Recognition v2 on Android
• JapaneseTextRecognizerOptions

The main difference

The experiment doesn't replace Codename One OCR with another OCR engine. Both Android tests use ML Kit.

The observable difference is essentially:

// Current CN1 Android implementation
TextRecognition.getClient(TextRecognizerOptions.DEFAULT_OPTIONS);

versus:

// Japanese-capable ML Kit configuration
TextRecognition.getClient(
        new JapaneseTextRecognizerOptions.Builder().build());

plus the corresponding script-specific dependency:

com.google.mlkit:text-recognition-japanese:16.0.1

So, at least on Android, the underlying backend already has the required capability. What appears to be missing in the portable CN1 API is a way to request the recognition script/language so that the Android adapter and dependency selection can choose the matching ML Kit recognizer/model.

[shai-almog]

This is implemented and should be a in the upcoming update.