Client Side Tool Calling with the OpenAI WebRTC Realtime API

This project is a fork of this repo by my wonderful teammate Craig Dennis. It's a Cloudflare Workers app using Hono to relay the OpenAI Realtime API over WebRTC. The main file is static, then the resulting files are dynamic.

It edits Craig's fork to fill out a form on a webpage via your voice. When you submit the form, Llama-3.2-3b-instruct hosted on Cloudflare Workers AI generates gift recommendations for someone according to how you answer the questions, and then saves the gift recommendation and people can rate and thus rank the gift recommendations.

It does create an ephemeral token and create a data channel.

Develop

Copy .dev.vars.example to .dev.vars and fill out your OpenAI API Key.

Install your dependencies

npm install

Run local server

npm run dev

Deploy

Upload your OpenAI API key (secret)

npx wrangler secret put OPENAI_API_KEY

npm run deploy

The hand (in Craig's app) is a HiWonder AI Hand. AI and I reverse-engineered the mobile app to make it work over Bluetooth, see the code in hand.js