Highlight scan barcodes for better UX

[qboisson]

While scanning barcodes, it is possible to retrieve a barcode object with either boundingBox for Android or cornerPoints for iOS.

With these properties, it seems possible to get the right positioning of the barcode in order to highlight it for better user experience.
For instance, while scanning multiple barcodes, it can be useful to add an emoji or show the borders of the barcodes already scanned.

The issue is that I am not able to display correctly the borders of scanned barcodes.
I am trying to adapt boundingBox and cornerPoints to use these values in a more classic way to display absolute elements with left and top properties but it does not fit perfectly with the real barcode borders.
Is there a way to use boundingBox or cornerPoints correctly ?

See below an example to reproduce this issue and an example of cornerPoints value.

import React, {useEffect, useState} from 'react';
import {
  View,
  Text,
  StyleSheet,
  PixelRatio,
  Platform,
} from 'react-native';
import {Camera, useCameraDevices} from 'react-native-vision-camera';

import {Colors} from 'react-native/Libraries/NewAppScreen';
import {BarcodeFormat, useScanBarcodes} from 'vision-camera-code-scanner';

const styles = StyleSheet.create({
  outer: {
    ...StyleSheet.absoluteFillObject,
    backgroundColor: Colors.darker,
    flex: 1,
  },
});

const SquareFromCoord = ({barcode}) => {
  let bottom;
  let left;
  let right;
  let top;

  const pixelRatio = PixelRatio.get();
  if (Platform.OS === 'ios') {
    const xArray = barcode.cornerPoints.map(corner => parseFloat(corner.x));
    const yArray = barcode.cornerPoints.map(corner => parseFloat(corner.y));

    left = Math.min(...xArray) / pixelRatio;
    right = Math.max(...xArray) / pixelRatio;
    bottom = Math.max(...yArray) / pixelRatio;
    top = Math.min(...yArray) / pixelRatio;
  } else {
    // Other similar logic for Android
  }

  return (
    <View
      style={{
        backgroundColor: 'rgba(0, 255, 131, 0.8)',
        borderRadius: 10,
        height: bottom - top,
        opacity: 0.8,
        width: right - left,
        position: 'absolute',
        left,
        top,
      }}
    />
  );
};

export const RnCamera = () => {
  const devices = useCameraDevices();
  const device = devices.back;

  const [frameProcessor, barcodes] = useScanBarcodes([BarcodeFormat.QR_CODE]);
  
  const [permissions, setPermissions] = useState(false);

  const handlePermission = async () => {
    const cameraPermission = await Camera.getCameraPermissionStatus();
    if (cameraPermission !== 'authorized') {
      const askPermissionRes = await Camera.requestCameraPermission();
      setPermissions(askPermissionRes === 'authorized');
    } else {
      setPermissions(true);
    }
  };

  useEffect(() => {
    handlePermission();
  }, []);

  if (!device || !permissions) {
    return <Text>Loading</Text>;
  }

  return (
    <View style={styles.outer}>
      <Camera
        style={StyleSheet.absoluteFill}
        device={device}
        isActive={true}
        enableZoomGesture
        frameProcessor={frameProcessor}
        frameProcessorFps={5}
      />
      {barcodes.map((barcode, idx) => (
        <SquareFromCoord key={idx} barcode={barcode} />
      ))}
    </View>
  );
};

Example of cornerPoints while scanning a QR code with an iPad Air 2 (window dimensions = 677:335) :
"cornerPoints": [{"x": 209.60528564453125, "y": 647.1997680664062}, {"x": 331.0139465332031, "y": 655.4308471679688}, {"x": 330.98651123046875, "y": 780.6898193359375}, {"x": 209.5778350830078, "y": 772.458740234375}]

[kirill3333]

@qboisson I've changed it a little and it looks very close to reality

const WINDOW_HEIGHT = Dimensions.get('window').height;
const WINDOW_WIDTH = Dimensions.get('window').width;

...
const frameProcessor = useFrameProcessor(frame => {
    'worklet';
    const detectedBarcodes = scanBarcodes(frame, [BarcodeFormat.ALL_FORMATS]);
    const barcode = detectedBarcodes[0];
    if (barcode) {
      const cornerPoints = barcode.cornerPoints;
      const xRatio = frame.width / WINDOW_WIDTH;
      const yRatio = frame.height / WINDOW_HEIGHT;
      if (Platform.OS === 'ios') {
        const xArray = cornerPoints.map(corner => parseFloat(corner.x));
        const yArray = cornerPoints.map(corner => parseFloat(corner.y));
        const resultPoints = {
          left: Math.min(...xArray) / xRatio,
          right: Math.max(...xArray) / xRatio,
          bottom: Math.max(...yArray) / yRatio,
          top: Math.min(...yArray) / yRatio,
        };
        runOnJS(setPointsOfInterest)(resultPoints);
      } else {
        // Other similar logic for Android
      }
      runOnJS(setBarcodes)(detectedBarcodes);
    } else {
      runOnJS(setPointsOfInterest)(null);
    }
  }, []);
...

[saajuck]

It works realy great, thanks a lot @kirill3333 !
i'm surprised that this relation between frame and windows dimension isn't described in any documentation

[kirill3333]

@saajuck this is because your screen size is different from the size of the image you are sending to process. I am pretty sure the frame processor shouldn't know about screen size.

[toxicity1985]

Hello,

Is it possible to get an example. I was trying to use your code but it doesn't work on my side.

[anagovitsyn]

kirill3333

Works on ios, but on android there's some offset from actual qrcode, why?

[mgcrea]

Unfortunately it's more complex than the current given solutions as it actually depends on the camera view size (that can be equal to the window when in fullscreen), and there is also some cropping after the resize that you have to take into account to setup proper offsets.

Currently working on a solution that covers all cases, @rodgomesc would you be interested in a PR adding this highlight feature?

[zacdemi]

Would be great! @mgcrea

[oscar-b]

@mgcrea Any news on this? Also, wouldn't it make sense to use animated components from Reanimated for the bounding boxes?

[ChauVV]

For android and ios
i use fork of this lib: https://github.com/rematocorp/vision-camera-code-scanner.git
to get frameWidth, frameHeight
and in my javascript:

const WINDOW_HEIGHT = Dimensions.get('window').height;
const WINDOW_WIDTH = Dimensions.get('window').width;
const SCAN_WIDTH = 200;

const scanBound = {
  height: SCAN_WIDTH,
  width: SCAN_WIDTH,
  x: (WINDOW_WIDTH - SCAN_WIDTH) / 2,
  y: (WINDOW_HEIGHT - SCAN_WIDTH) / 2,
};

const aabb = (barcode, frameWidth, frameHeight) => {
  try {
    const { cornerPoints } = barcode;

    const frWidth = Math.min(frameHeight, frameWidth);
    const frHeight = Math.max(frameHeight, frameWidth);

    const xRatio = frWidth / WINDOW_WIDTH;
    const yRatio = frHeight / WINDOW_HEIGHT;

    const xArray = cornerPoints.map(corner => parseFloat(corner.x));
    const yArray = cornerPoints.map(corner => parseFloat(corner.y));

    const left = Math.min(...xArray);
    const right = Math.max(...xArray);
    const bottom = Math.max(...yArray);
    const top = Math.min(...yArray);

    const x = left / xRatio;
    const y = top / yRatio;
    const w = (right - left) / xRatio;
    const h = (bottom - top) / xRatio;

    return (
      cornerPoints.length === 4 &&
      scanBound.x <= x &&
      scanBound.x + scanBound.width >= x + w &&
      scanBound.y <= y &&
      scanBound.y + scanBound.height >= y + h
    );
  } catch (error) {
    return false;
  }
};