react-native-vision-camera-mlkit

📦 A Google ML Kit frame processor plugin for react-native-vision-camera.

This plugin provides a simple way to use various ML Kit Vision APIs in your React Native App's Frame processor.

⚠️ This plugin is still in development and not yet ready for iOS.

🧵 Vision APIs

Video and image analysis APIs to label images and detect barcodes, text, faces, and objects.

• Barcode scanning Scan and process barcodes. Supports most standard 1D and 2D formats.
• Face detection Detect faces and facial landmarks.
• Face mesh detection Detect face mesh info on close-range images.
• Text recognition v2 Recognize and extract text from images.
• Image labeling Identify objects, locations, activities, animal species, products, and more. Use a general-purpose base model or tailor to your use case with a custom TensorFlow Lite model.
• Object detection and tracking Localize and track in real time one or more objects in the live camera feed.
• Digital ink recognition Recognizes handwritten text and handdrawn shapes on a digital surface, such as a touch screen. Recognizes 300+ languages, emojis and basic shapes.
• Pose detection Detect the position of the human body in real time.
• Selfie segmentation Separate the background from users within a scene and focus on what matters.
• Subject segmentation Separate subjects (people, pets, or objects) from the background in a picture.
• Document scanner Digitize physical documents from pictures.

🚀 Getting Started

🚨 Required Dependencies

Ensure you have installed the required packages before installing this plugin.

Package	Version
react-native-vision-camera	>=4.0.1
react-native-worklets-core	>=1.2.0

Follow the installation instructions for each package.

💻 Installation

To install the plugin, run:

npm install react-native-vision-camera-mlkit
# or
yarn add react-native-vision-camera-mlkit

🪝 Hooks

useBarcodeScanner (Barcode scanning)

  const { barcodeScanner } = useBarcodeScanner();

  const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet';

      runAsync(frame, () => {
        'worklet';

        const barcodes = barcodeScanner(frame);
        console.log(barcodes);
      });
    },
    [barcodeScanner]
  );

useImageLabeler (Image labeling)

  const { imageLabeler } = useImageLabeler();

  const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet';

      runAsync(frame, () => {
        'worklet';

        const labels = imageLabeler(frame);
        console.log(labels);
      });
    },
    [imageLabeler]
  );

useObjectDetector (Object detection and tracking)

  const { objectDetector } = useObjectDetector();

  const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet';

      runAsync(frame, () => {
        'worklet';

        const objects = objectDetector(frame);
        console.log(objects);
      });
    },
    [objectDetector]
  );

useTextRecognizer (Text recognition v2)

  const { textRecognizer } = useTextRecognizer();

  const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet';

      runAsync(frame, () => {
        'worklet';

        const text = textRecognizer(frame);
        console.log(text);
      });
    },
    [textRecognizer]
  );