Skip to content
This repository has been archived by the owner. It is now read-only.
Real time image classification with React Native and Tensorflow lite.
Java Objective-C C# JavaScript Ruby
Branch: master
Clone or download
Latest commit e299927 Dec 25, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github/ISSUE_TEMPLATE Update question.md Nov 19, 2018
android Updated error handling in case of failure Dec 24, 2018
docs Updated js source and docs Dec 22, 2018
examples/basic chore(example): remove unused methods and properties (#2009) [skip ci] Dec 17, 2018
ios fix(rct-camera): crash when metadata.stringValue is nil (#2016) Dec 19, 2018
postinstall_project Seblau/fix postinstall pbxproj (#1961) Dec 3, 2018
src Now attempts to pass data as react props Dec 23, 2018
types Updated js source and docs Dec 22, 2018
windows Windows: added README & fixed property name (#1966) Dec 3, 2018
.eslintrc docs(rn-camera): add rncamera-example without face detection (#1979) … Dec 11, 2018
.flowconfig chore(lint): fix lint to make ci work Apr 15, 2018
.gitattributes disable git large files Feb 1, 2018
.gitignore chore: upgrade prettier and autoformat typescript files upon commit (… Dec 12, 2018
.npmignore fix: exclude android build artifacts (#2001) [skip ci] Dec 17, 2018
.prettierrc docs(rn-camera): add rncamera-example without face detection (#1979) … Dec 11, 2018
CHANGELOG.md chore(changelog): add link to github releases [skip ci] Nov 12, 2018
LICENSE Initial commit Apr 1, 2015
README.md Updated readme Dec 26, 2018
THIRD-PARTY-LICENSES rename expo to simple RN, add js files, and export them inside the sa… Jan 1, 2018
circle.yml fix: exclude android build artifacts (#2001) [skip ci] Dec 17, 2018
commitlint.config.js chore(semantic-release): setup semantic-release for automated publish… Nov 12, 2018
package.json 0.0.3 Dec 26, 2018
react-native-camera.podspec update CocoaPod subspecs versions Oct 20, 2018
yarn.lock fix(rn-camera): remove lodash dependency (#1996) Dec 14, 2018

README.md

Related blog article

For more information, see react-native-camera. All react-native-camera props/features should also work with this.

Real time image classification with React Native

Earlier attempts at Image classification over React Native involved sending image data to the model classifier by sending the image over the bridge or storing the image to disk and accessing the image on the native side. Here's an attempt at live image classification by processing from the camera feed on the native side and getting the output as a byte stream on the JS side using the react-native-camera-tflite library.

Huge shout-out to the people over at react-native-camera. This is essentially just a fork of their awesome work.

Note: This is currently developed only for Android but could be implemented for iOS. (See here for a CoreML implementation on iOS).

To start, let's create an empty react native project:

react-native init mobilenetapp
cd mobilenet-app

Let's add our dependencies:

npm i react-native-camera-tflite
react-native link react-native-camera-tflite

Follow the install instructions (for android. Same as react-native-camera):

  1. Insert the following lines inside the dependencies block in android/build.gradle:
    ...
    allprojects {
        repositories {
            maven { url "[https://jitpack.io](https://jitpack.io)" }  
            maven { url "[https://maven.google.com](https://maven.google.com)" }

    ...
    ...
    ext {
        compileSdkVersion           = 26
        targetSdkVersion            = 26
        buildToolsVersion           = "26.0.2"
        googlePlayServicesVersion   = "12.0.1"
        supportLibVersion           = "27.1.0"
    }
  1. Insert the following lines inside android/app/build.gradle

    android { ... aaptOptions { noCompress "tflite" noCompress "lite" } ...

Now let's use the download our model file from here, decompress it, and copy over the mobilenet_v1_1.0_224_quant.tflite file over to our project.

    mkdir -p ./android/app/src/main/assets
    cp mobilenet_v1_1.0_224_quant.tflite ./android/app/src/main/assets

Add this file to your project root directory as Output.json

Replace the content of App.js in your project root directory with the following:

    import React, {Component} from 'react';
    import {StyleSheet, Text, View } from 'react-native';
    import { RNCamera } from 'react-native-camera-tflite';
    import outputs from './Output.json';
    import _ from 'lodash';

    let _currentInstant = 0;

    export default class App extends Component {
      constructor(props) {
        super(props);
        this.state = {
          time: 0,
          output: ""
        };
      }

    processOutput({data}) {
        const probs = _.map(data, item => _.round(item/255.0, 0.02));
        const orderedData = _.chain(data).zip(outputs).orderBy(0, 'desc').map(item => [_.round(item[0]/255.0, 2), item[1]]).value();
        const outputData = _.chain(orderedData).take(3).map(item => `${item[1]}: ${item[0]}`).join('\n').value();
        const time = Date.now() - (_currentInstant || Date.now());
        const output = `Guesses:\n${outputData}\nTime:${time} ms`;
        this.setState(state => ({
          output
        }));
        _currentInstant = Date.now();
      }
      
      render() {
        const modelParams = {
          file: "mobilenet_v1_1.0_224_quant.tflite",
          inputDimX: 224,
          inputDimY: 224,
          outputDim: 1001,
          freqms: 0
        };
        return (
          <View style={styles.container}>
            <RNCamera
                ref={ref => {
                    this.camera = ref;
                  }}
                style = {styles.preview}
                type={RNCamera.Constants.Type.back}
                flashMode={RNCamera.Constants.FlashMode.on}
                permissionDialogTitle={'Permission to use camera'}
                permissionDialogMessage={'We need your permission to use your camera phone'}
                onModelProcessed={data => this.processOutput(data)}
                modelParams={modelParams}
            >
              <Text style={styles.cameraText}>{this.state.output}</Text>
            </RNCamera>
          </View>
        );
      }
    }

    const styles = StyleSheet.create({
      container: {
        flex: 1,
        flexDirection: 'column',
        backgroundColor: 'black'
      },
      preview: {
        flex: 1,
        justifyContent: 'center',
        alignItems: 'center'
      },
      cameraText: {
        color: 'white',
        fontSize: 18,
        fontWeight: 'bold'
      }
    });

We're done! Run your app with the following command.

    react-native run-android

Image Classification FTW!Image Classification FTW!

To convert this to a hotdog not-hotdog app, just replace the processOutput function above with the following:

    processOutput({data}) {
      const isHotDogProb = data[935];
      const isHotDog = isHotDogProb > 0.2 ? "HotDog" : "Not HotDog";
      const time = Date.now() - (_currentInstant || Date.now());
      const output = `${isHotDog}\nTime:${time} ms`;
      this.setState(state => ({
       output
      }));
     _currentInstant = Date.now();
    }

Run your app with the following command.

    react-native run-android

It's HotDogIt’s HotDog

Jian Yang would be proud :)

This project has a lot of rough edges. I hope to clean up this up a lot more in the coming days. The rest of the features are the same as react-native-camera.

Links: Github Demo App npm

You can’t perform that action at this time.