Switch branches/tags
Nothing to show
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


Cozmo 4th of July with ARKit

See what's going on inside Cozmo's little head with ARKit.

We did an experiment with the latest release of ARKit to bring Cozmo's imagination to life as he gears up for July 4th Independence Day celebrations.

This project contains a Python server that uses UDP to communicate with an ARKit Unity app. The server is responsible for communicating with the app and controlling Cozmo's actions using the Cozmo Python SDK.

The Unity project lets you see Cozmo's imagination using the ARKit Unity plugin.

Cozmo 4th of July with ARKit

Cozmo ARKit Server

Getting Started

Install the Cozmo SDK

Before running this experience, you'll want to make sure you have the latest version of the Cozmo SDK. If you don't have the SDK installed, start by reading the instructions.

Install Dependencies

$ pip3 install -r requirements.txt

Obtain Device IP Address

  • Make sure the device and the machine running server are on the same wi-fi network.
  • On your device open Settings > Wi-Fi and tap on the network name and note the "IP Address".
  • Open "lib/config.py" and update the IP_ADDRESS value with your device's IP address.
  • Make sure the PORT matches the port number in the Unity project (set to 8000 by default).

Run the Server

$ python3 server.py

Cozmo ARKit for Unity

Minimum Software Requirements

Unity Plug-ins

Unity Build Instructions

  • Install all software
  • Open the Unity project
  • Download and import the required plug-ins from the asset store
  • Open "/Assets/Scenes/Main.unity"
  • Play the scene in the editor to test (use keyboard keys 1-4 to test animations without the server)
  • Build the project for iOS

Xcode Build Instructions

  • Open the Xcode project generated by your Unity build
  • Sign the product either with your mobile provisioning profile or by having Xcode sign it automatically for you
  • Open the target settings
  • Go to "Build Phases" > "Link Binary with Library", then add "libicucore.tbd"
  • Go to "Build Settings" > "Enable Bitcode" and set the value to "No" for all targets
  • Make sure your iPhone 6S or later is connected and targeted, then build or run from Xcode

Using the App

  • Clear a table
  • Set one of Cozmo's blocks at the center of the table
  • Place Cozmo on the far side of the table away from you, facing his block (he should now have both you and his block in his view)
  • Run the server + Cozmo app
  • Open the AR app
  • Hold your device about a foot above the table, aiming at Cozmo's block
  • Pan and tilt the device around until a blue rectangle appears on the table
  • Continue panning until the recatngle grows into roughly a 2 - 4' wide square
  • Tap your device's screen where the block appears
  • The scene will appear, centered around the block
  • If you entered your server information correctly and the server is running, then your virtual scene should be synchronized with Cozmo's real life actions
  • If tracking drifts, tap the block again to re-center the scene

Developer Notes

Use this project as a base to create your own scenes for Cozmo. Follow the comments in the code for insights into how it all works. Please share what you make!