diff --git a/README.md b/README.md index eb551f3..22180d0 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,11 @@ -# openai api mock +# OpenAI API mock / proxy Worker -this project aims to convert / proxy Cloudflare Workers AI responses to OpenAI API compatible responses so Workers AI models can be used with any OpenAI / ChatGPT compatible client +This project aims to convert/proxy Cloudflare Workers AI responses to OpenAI API compatible responses so that [Cloudflare Workers AI](https://developers.cloudflare.com/workers-ai/) models can be used with any OpenAI/ChatGPT compatible client. -- supports streaming and non-streaming responses -- rewrites the different models 'gpt-3' and 'gpt-4' to use `@cf/meta/llama-2-7b-chat-fp16` -- if the openAI client can be configured to provide other model names, just put in the cloudflare model id instead of gpt-4 +- Supports streaming and non-streaming responses +- Rewrites *default* models such as `gpt-3` and `gpt-4` to use `@cf/meta/llama-3-8b-instruct` +- If the OpenAI client can be configured to use other model names, simply replace `gpt-4` with the Cloudflare model ID +- Here's a list of all [Cloudflare Workers AI models](https://developers.cloudflare.com/workers-ai/models/) ## installation @@ -14,17 +15,20 @@ this project aims to convert / proxy Cloudflare Workers AI responses to OpenAI A 2. clone this repo 3. run `npm run deploy` -after the script has been deployed, you'll get an URL which you can use as your OpenAI API endpoint for other applications, something like this: +after the script has been deployed, you'll get an URL which you can use as your OpenAI API endpoint for other applications, something like this: `https://openai-api.foobar.workers.dev` -``` -https://openai-api.foobar.workers.dev -``` +## examples -## use with llm +### use with llm -- [read and install everything about llm here](https://llm.datasette.io/) +I mainly created this project to make it work with the awesome [LLM project]((https://llm.datasette.io/)) from [Simon Willison](https://simonwillison.net/) -create this file `~/Library/Application\ Support/io.datasette.llm/extra-openai-models.yaml`: +- [go ahead and read everything about LLM here](https://llm.datasette.io/) +- [how to install LLM](https://llm.datasette.io/en/stable/setup.html) +- since LLM can work with [OpenAI-compatible models](https://llm.datasette.io/en/stable/openai-models.html#adding-more-openai-models), we're adding our OpenAI API proxy like this: + +1. find the directory of your llm configuration: `dirname "$(llm logs path)"` +2. create this file: `vi ~/Library/Application\ Support/io.datasette.llm/extra-openai-models.yaml` ```yaml - model_id: cloudflare @@ -32,14 +36,25 @@ create this file `~/Library/Application\ Support/io.datasette.llm/extra-openai-m api_base: 'https://openai-api.foobar.workers.dev/' ``` +you can also add multiple models there: + +```yaml +- model_id: cfllama2 + model_name: "@cf/meta/llama-2-7b-chat-fp16" + api_base: "https://openai-api.foobar.workers.dev" +- model_id: cfllama3 + model_name: "@cf/meta/llama-3-8b-instruct" + api_base: "https://openai-api.foobar.workers.dev" +``` + use it with streaming (recommended): ```shell -llm chat -m cloudflare +llm chat -m cfllama3 ``` use it without streaming: ```shell -llm chat --no-stream -m cloudflare +llm chat --no-stream -m cfllama3 ```