Skip to content
Switch branches/tags
Go to file
This branch is 1 commit ahead, 18 commits behind IBM:master.

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Visual Recognition for Anki Cozmo with TensorFlow

The Anki Cozmo robot can recognize faces and objects like Cozmo's Power Cubes which have markers on them. This project contains sample code so that Cozmo can recognize other types of objects via TensorFlow.

Watch the video and check out the slides to see how Cozmo can recognize three different toys:




The training is done via TensorFlow and a retrained MobileNet model on Kubernetes.

alt text

The classification is done via Tensorflow running in an OpenWhisk function.

alt text

For more details check out the blog entries from Ansgar and me:


Install the Cozmo SDK.

Get a free IBM Cloud lite account.

Install the IBM Cloud/Bluemix CLI.

Install Docker.

Create a free/lite Kubernetes cluster.

Install the Kubernetes CLI.

1. Take Pictures

Take pictures of an object by invoking these commands and circling Cozmo around the object for 10 seconds. Replace 'deer' with a name for your object:

$ git clone
$ cd visual-recognition-for-cozmo-with-tensorflow/1-take-pictures
$ python3 deer

2. Upload Pictures

Create an IBM Object Storage instance. Choose the lite Swift option. Read Ansgar's blog for details.

Copy/remember the IBM Object Storage credentials: ‘region’, ‘projectId’, ‘userId’ and ‘password’. Paste them in

Invoke these commands:

$ cd visual-recognition-for-cozmo-with-tensorflow/2-upload-pictures
$ pip3 install python-swiftclient
$ pip3 install python-keystoneclient
$ python3

3. Train the Model

Paste the values of ‘region’, ‘projectId’, ‘userId’ and ‘password’ in train.yml.

Replace 'nheidloff' with your DockerHub name and run these commands:

$ cd visual-recognition-for-cozmo-with-tensorflow/3-train
$ docker build -t nheidloff/tensorflow-openwhisk-train-cozmo:latest .
$ docker push nheidloff/tensorflow-openwhisk-train-cozmo:latest
$ bx login -a
$ bx cs cluster-config mycluster
$ export KUBECONFIG=/Users/nheidlo.....
$ kubectl apply -f train.yml 

After this you should see the files 'retrained_graph_cozmo.pb' and 'retrained_labels_cozmo.txt' in the 'tensorflow' container in IBM Object Storage.

Read Ansgar's blog for more details.

4. Deploy the Model to OpenWhisk

Paste the values of ‘region’, ‘projectId’, ‘userId’ and ‘password’ in

Replace 'nheidloff' with your DockerHub name and run these commands:

$ cd visual-recognition-for-cozmo-with-tensorflow/4-classify
$ docker build -t nheidloff/tensorflow-openwhisk-classifier-cozmo:latest .
$ docker push nheidloff/tensorflow-openwhisk-classifier-cozmo:latest
$ cd visual-recognition-for-cozmo-with-tensorflow/4-classify/openwhisk-api
$ bx login -a
$ bx target -o <your-organization> -s <your-space>
$ bx plugin install Cloud-Functions -r Bluemix
$ wsk package create visualRecognitionCozmo
$ wsk action create visualRecognitionCozmo/tensorflow-classify --docker nheidloff/tensorflow-openwhisk-classifier-cozmo:latest
$ npm install
$ sh ./
$ wsk action create --sequence visualRecognitionCozmo/classifyAPI visualRecognitionCozmo/classifyImage,visualRecognitionCozmo/tensorflow-classify --web raw

5. Test the Model via the Web Application

In the OpenWhisk web application choose your sequence and open 'Additional Details'. From there copy the URL into the clipboard. Create a new file '.env' in the '5-test-in-web-app' directory. See .env-template for an example. Paste the URL in this file.

From the directory 'web-app' run these commands:

$ cd visual-recognition-for-cozmo-with-tensorflow/5-test-in-web-app
$ npm install
$ npm start

Open the web application via http://localhost:3000/.

Optionally: In order to deploy the application to the IBM Cloud, change the application name in manifest.yml to something unique and run these commands:

$ bx login -a
$ bx target -o <your-organization> -s <your-space>
$ cf push

Open the web application via

6. Test Visual Recognition with Cozmo

Place your object(s) in a circle around Cozmo and run these commands. Replace 'deer' with a name for your object:

$ cd visual-recognition-for-cozmo-with-tensorflow/6-play-with-cozmo
$ python3 deer


Visual Recognition for Anki Cozmo with TensorFlow




No releases published


No packages published