how to improve the QR code recognition rate.

[jinsw1081]

i use zxing.unity.dll

I'm making a mobile QR code recognition AR program in Unity.
Device used S10+

However, the qrcode recognition is not very good.
For example, in the basic application camera, QR is recognized, but if Zxing.unity.dll is used in Unity, QR is not recognized. The closer you get to it, the more it will be recognized.
Do you know how to increase the recognition rate by any chance?

Here's how I came up with it.

1. Camera focus setting

2. Accurate QR code printing

3. Increase the QR size itself

All of these methods were useless.

How can I increase the recognition rate?

[micjahn]

Do you use the newest version of ZXing.Net?
Can you provide the source code of your zxing integration and a sample dump of an image which you try to decode?

[jinsw1081]

i use Latest version

this QRCode

i use this cord

void OnCameraFrameReceived(ARCameraFrameEventArgs eventArgs)
    {
        if ((Time.frameCount % 15) == 0)
        { 
//You can set this number based on the frequency to scan the QRCode
            XRCameraImage image;
            if (aRCamera.TryGetLatestImage(out image))
            {
                StartCoroutine(ProcessQRCode(image));
                image.Dispose();
            }
        }
    }
  

    //Asynchronously Convert to Grayscale and Color : https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@1.0/manual/cpu-camera-image.html
    IEnumerator ProcessQRCode(XRCameraImage image)
    {
        // Create the async conversion request
        var request = image.ConvertAsync(new XRCameraImageConversionParams
        {
            inputRect = new RectInt(0, 0, image.width, image.height),
            outputDimensions = new Vector2Int(image.width / 2, image.height / 2),
            // Color image format
            outputFormat = TextureFormat.RGB24,
            // Flip across the Y axis
            //  transformation = CameraImageTransformation.MirrorY
        });
        while (!request.status.IsDone())
            yield return null;
        // Check status to see if it completed successfully.
        if (request.status != AsyncCameraImageConversionStatus.Ready)
        {
            // Something went wrong
            Debug.LogErrorFormat("Request failed with status {0}", request.status);
            // Dispose even if there is an error.
            request.Dispose();
            yield break;
        }
        // Image data is ready. Let's apply it to a Texture2D.
        var rawData = request.GetData<byte>();
        // Create a texture if necessary
        if (arCameraTexture == null)
        {
            arCameraTexture = new Texture2D(
            request.conversionParams.outputDimensions.x,
            request.conversionParams.outputDimensions.y,
            request.conversionParams.outputFormat,
            false);
        }
        // Copy the image data into the texture
        arCameraTexture.LoadRawTextureData(rawData);
        arCameraTexture.Apply();
        byte[] barcodeBitmap = arCameraTexture.GetRawTextureData();
        LuminanceSource source = new RGBLuminanceSource(barcodeBitmap, arCameraTexture.width, arCameraTexture.height);
        //Send the source to decode the QRCode using ZXing
        if (true)
        { 
            //Check if a frame is already being decoded for QRCode. If not, get inside the block.
            doOnce = true; //Now frame is being decoded for a QRCode
                           //decode QR Code
            result = reader.Decode(source);

            if (result != null && result.Text != "")
            { //If QRCode found inside the frame
                
                text1.text= result.Text;
                
                // Get the resultsPoints of each qr code contain the following points in the following order: index 0: bottomLeft index 1: topLeft index 2: topRight
                //Note this depends on the oreintation of the QRCode. The below part is mainly finding the mid of the QRCode using result points and making a raycast hit from that pose.
                ResultPoint[] resultPoints = result.ResultPoints;
                ResultPoint a = resultPoints[1];
                ResultPoint b = resultPoints[2];
                ResultPoint c = resultPoints[0];
                Vector2 pos1 = new Vector2((float)a.X, (float)a.Y);
                Vector2 pos2 = new Vector2((float)b.X, (float)b.Y);
                Vector2 pos3 = new Vector2((float)c.X, (float)c.Y);
                Vector2 pos4 = new Vector2(((float)b.X - (float)a.X) / 2.0f, ((float)c.Y - (float)a.Y) / 2.0f);
                List<ARRaycastHit> aRRaycastHits = new List<ARRaycastHit>();
                //Make a raycast hit to get the pose of the QRCode detected to place an object around it.
                if (arRaycastManager.Raycast(new Vector2(pos4.x, pos4.y), aRRaycastHits, TrackableType.FeaturePoint) && aRRaycastHits.Count > 0)
                {
                    //To shift the object to a relative position by adding/subtracting a delta value, uncomment the below line.
                    //Instantiate an object at Hitpose found on the QRCode

                    //GameObject NewObjectToPlace = Instantiate(arObjectOnQRCode, aRRaycastHits[0].pose.position,Quaternion.identity);

                    //OR
                    // Use default position to place the object in front of the camera if the Hit Pose is not found. //You can uncomment the below code for this default behaviour
                    //defaultObjectPosition = Camera.main.ScreenToWorldPoint(new Vector3(Screen.width / 2, //Screen.height / 2, Camera.main.nearClipPlane));
                    //OR
                    //Reposition the Augmented Object by adding some delta
                    //NewObjectToPlace.transform.position = new //Vector3(NewObjectToPlace.transform.position.x + xDelta, //NewObjectToPlace.transform.position.y, NewObjectToPlace.transform.position.z);
                }
                else
                {
                    doOnce = false; //Continue processing the next frame to decoded for a QRCode if hit not found
                }
            }
            else
            {
                doOnce = false;  //QRCode not found in the frame. Continue processing next frame for QRCode
            }
        }

[micjahn]

I need the content of the variables "barcodeBitmap", "arCameraTexture.width" and "arCameraTexture.height" for further investigation.

[jinsw1081]

The little camera rectangle is what makes the arCameraTexture visible on the Unity screen
barcodeBitmap is the raw data read from the texture.

[micjahn]

Sorry, that doesn't help. I need a binary dump file of the contents of the "barcodeBitmap" variable.
Something like this:

...
        byte[] barcodeBitmap = arCameraTexture.GetRawTextureData();
System.IO.File.WriteAllBytes("<dump file name here>.bin", barcodeBitmap);
        LuminanceSource source = new RGBLuminanceSource(barcodeBitmap, arCameraTexture.width, arCameraTexture.height);
...

[jinsw1081]

thank you for the reply.
After working on an urgent project for a while, I came back to this project.
arCameraTexture
width is 640
height is 480
barcodeBitmap is

This is the extracted file

SSSSS.zip
This is the image when it is not recognized

[micjahn]

The following image is "seen" by the decoder. It represents the luminance values. The qr code seems to be too small and too blurry. Is there a chance that you can use a higher camera resolution? And it is rotated by 90 degrees. Perhaps the image orientation isn't correct.

[jinsw1081]

1920x1080.zip
I changed the resolution by saying it might be a resolution problem above.
the image in the code, but it's embarrassing the image rotate.
However, when I try to add image rotation, I still receive 1920x1080 images every frame%10, so performance suffers. If I rotate this, it seems to be even more difficult.

The QR code recognition doesn't seem to have improved. Do you know how?

And is it possible to distinguish up, down, left and right through the QR code?

ex) If you scan the QR code on the floor, it will be printed upside down
QR <- not phone
phone -> QR

For reference, when I say that QR recognition doesn't work well, it doesn't mean that it doesn't work well at a close distance, i.e. within 1m, but it has to be done at 2-3m.

As mentioned in the first question, in the basic built-in camera application, 3~4m is recognized.

[micjahn]

I'm not sure if ZXing.Net is a good solution for your use case. ZXing (the java version, which ZXing.Net is based on) in general was developed for mobile phones. In most cases, the user holds the phone near to the barcode. That scenario is the main use case. I fear you will never fully resolve any issues which you actually have with long-distance scanning.

Btw. in the current binary dump I can't visually find any QR code.

If you activate the AutoRotate option with the BarcodeReader you will get some information about the barcode rotation within the ResultMetadata property of the result object (ResultMetadataType.ORIENTATION).

[jinsw1081]

Thank you so much for your reply.

           BarcodeReader barcodeReader = new BarcodeReader();
            barcodeReader.AutoRotate = true;
           IBarcodeReader    reader = new BarcodeReader();


           object objValue;
            if (result.ResultMetadata.ContainsKey(ResultMetadataType.ORIENTATION))
                ; //this is breakpoint

            result.ResultMetadata.TryGetValue(ResultMetadataType.ORIENTATION,out objValue);

As you said, I tried to access the direction, but it doesn't hit a breakpoint.
Do you know why?

[micjahn]

I'm not sure. Perhaps the optimizer threw away the empty code block.
Your code snippet seems to be incomplete. I don't see the Decode call.

[jinsw1081]

public class ARQRCodeCS : MonoBehaviour
{
IBarcodeReader reader; //QRCode reading library
void Start()
{
//Get the ZXing Barcode/QRCode reader
BarcodeReader barcodeReader = new BarcodeReader();
barcodeReader.AutoRotate = true;
reader = new BarcodeReader();

}

IEnumerator ProcessQRCode(XRCpuImage image)
{
    // Create the async conversion request
    var request = image.ConvertAsync(new  XRCpuImage.ConversionParams 
    {
        inputRect = new RectInt(0, 0, image.width, image.height),
        outputDimensions = new Vector2Int(image.width , image.height ),
        // Color image format
        outputFormat = TextureFormat.RGB24,
        // Flip across the Y axis
    });
    while (!request.status.IsDone())
        yield return null;
    // Check status to see if it completed successfully.
    if (request.status != XRCpuImage.AsyncConversionStatus.Ready)
    {
        // Something went wrong
        Debug.LogErrorFormat("Request failed with status {0}", request.status);
        // Dispose even if there is an error.
        request.Dispose();
        yield break;
    }
    // Image data is ready. Let's apply it to a Texture2D.
    var rawData = request.GetData<byte>();
    // Create a texture if necessary
    if (arCameraTexture == null)
    {
        arCameraTexture = new Texture2D(
        request.conversionParams.outputDimensions.x,
        request.conversionParams.outputDimensions.y,
        request.conversionParams.outputFormat,
        false);
    }
    // Copy the image data into the texture
    arCameraTexture.LoadRawTextureData(rawData);
    arCameraTexture.Apply();
 
    byte[] barcodeBitmap = arCameraTexture.GetRawTextureData();


    LuminanceSource source = new RGBLuminanceSource(barcodeBitmap, arCameraTexture.width, arCameraTexture.height);
    //Send the source to decode the QRCode using ZXing

    source.rotateCounterClockwise();
    source.rotateCounterClockwise();
    source.rotateCounterClockwise();

    if (true)
    {
        string str;
        //Check if a frame is already being decoded for QRCode. If not, get inside the block.
        inScreen = true; //Now frame is being decoded for a QRCode
                       //decode QR Code
        result = reader.Decode(source);
        

         if (result?.ResultMetadata != null)
        {
            string ecLevel = result.ResultMetadata[ZXing.ResultMetadataType.ORIENTATION].ToString();
            string stt = null;
            var resultMetadataTypes = result.ResultMetadata.Keys;
            foreach (var v in resultMetadataTypes)
            {
               stt = stt+v.ToString();
            }

            text4.text = ecLevel.ToString();
        }
        if (result != null && result.Text != "")
        { 

            text1.text= result.Text;
          
         }
    }
    
    
}

}

ecLevel always 0

[micjahn]

Why do you do this within the method Start?

BarcodeReader barcodeReader = new BarcodeReader();
barcodeReader.AutoRotate = true;
reader = new BarcodeReader();

I think you should better write it like this here:

reader = new BarcodeReader();
reader.AutoRotate = true;

Am I wrong?

[jinsw1081]

That part I made a mistake. sorry.

BarcodeReader barcodeReader = new BarcodeReader();

    void Start()
    {
    

        //Get the ZXing Barcode/QRCode reader
        barcodeReader.AutoRotate = true;
        reader = barcodeReader;
        
          
    }

it, the same 0 comes out, can you figure out why?

[micjahn]

0 means that there was no rotation necessary.

[LoopIssuer]

Hi @micjahn
I have a question about one of your answers above: "The following image is "seen" by the decoder. "
Is there possibility to get this kind of image from the decoder in Unity using ZXing?

I have issue with Android, when one camera recognises QR code and other camera is not. So I would like to get some kind of debug, exception or some kind of 'decoder image' like you showed.

[micjahn]

@LoopIssuer
I dump the kind of image with an attached debugger. There is a method overload for ToString of the class BitMatrix which gives you a text representation of the binary image. I paste the text into notepad++ and zoom the view out until it is small enough.
Anyway, you can add a method to your code base which creates a similar image from the result of the HybridBinarizer class.
The implementation CommandLineDecoder has a method "dumpBlackPoint" which does similar things.