Frame AR Translation App

This app uses the Frame AR device to capture images and translate text using OpenAI's Vision API. This prototype was put together at Side Project Saturday.

Features

• Tap-to-Translate: Simply tap the side of Frame to capture an image
• Automatically sends the image to OpenAI's Vision API
• Displays the translation on the Frame display
• Supports any languages OpenAI can recognize and translate

Requirements

• Brilliant Labs Frame AR device
• Mise
• OpenAI API key

Installation

1. Clone this repository
2. Install the required packages:

mise install

3. Set up your OpenAI API key as an environment variable:

export OPENAI_API_KEY=your_api_key_here

Usage

1. Connect your Frame device via Bluetooth.
2. Run the application:

mise run translate

3. Once the app is running, you'll see "Tap to translate" on the Frame display.
4. Tap on the side of the Frame when pointing at text you want to translate.
   • The Frame will show a brief flash to confirm the tap was detected
   • "Taking photo..." will display while capturing
   • "Processing..." will display while the image is being analyzed
5. The translation result will appear on the Frame display.
6. Tap again to translate something else.

How It Works

The app consists of two parts:

1. Frame app (Lua) (lua/translate_app.lua):

   • Handles tap detection using Frame's IMU (motion sensor)
   • Captures photos when a tap is detected
   • Displays text and status messages

2. Host app (Python) (translator.py):

   • Manages communication with Frame
   • Processes photos captured by taps
   • Sends images to OpenAI with translation prompt
   • Returns translation results to Frame

Troubleshooting

Tap Detection Issues

If the app doesn't respond to taps:

• Make sure to tap firmly on the side of Frame
• Try tapping at different locations along the edge
• Check the console output for "Tap detected!" messages
• Restart the app if tap detection stops working

Other Issues

• Make sure your OpenAI API key is correctly set
• If translations fail, check your internet connection
• Ensure Frame is charged and connected via Bluetooth
• For better results, aim the Frame camera directly at clear, legible text
• Check the output images in the captures directory to make sure the text is legible.