Compatibility with chat UIs (BetterChatGPT)

[chitudorin]

I succesfully deployed the worker and confirmed it does work with OpenAI's API reference examples but running a fork of BetterChatGPT (in order to permit newer models) does not work.

Running log stream on my worker returns a 404 error and it looks like the request is an OPTIONS one, instead of GET.

[chand1012]

Okay that means that CORS is definitely the issue. That's definitely a bug, I'll see what I can do.

[chand1012]

Can you try deploying with the branch cors? I made a change that should fix that issue. Its the same branch as PR #4.

[chitudorin]

I tried deploying it but there are some build errors:

$ yarn deploy
yarn run v1.22.22
$ wrangler deploy index.js
 ⛅ wrangler 3.37.0
-------------------

✘ [ERROR] No matching export in "node_modules/itty-router/dist/itty-router.min.mjs" for import "createCors"

    index.js:1:17:
      1 │ import { Router, createCors, error, json } from 'itty-router';
        ╵                  ~~~~~~~~~~


✘ [ERROR] No matching export in "node_modules/itty-router/dist/itty-router.min.mjs" for import "error"

    index.js:1:29:
      1 │ import { Router, createCors, error, json } from 'itty-router';
        ╵                              ~~~~~


✘ [ERROR] No matching export in "node_modules/itty-router/dist/itty-router.min.mjs" for import "json"

    index.js:1:36:
      1 │ import { Router, createCors, error, json } from 'itty-router';
        ╵                                     ~~~~


✘ [ERROR] Build failed with 3 errors:

  index.js:1:17: ERROR: No matching export in "node_modules/itty-router/dist/itty-router.min.mjs"
  for import "createCors"
  index.js:1:29: ERROR: No matching export in "node_modules/itty-router/dist/itty-router.min.mjs"
  for import "error"
  index.js:1:36: ERROR: No matching export in "node_modules/itty-router/dist/itty-router.min.mjs"
  for import "json"

[chand1012]

Sorry about that! Forgot to update itty-router in the package file. Not sure why it built locally, but its fixed to be the latest version now.

[chitudorin]

It builds successfully now, but now it returns 500:

{
    "status": 500,
    "error": "Cannot read properties of undefined (reading 'AI')"
}

This is the request body:

{
   "messages":[
      {
         "role":"system",
         "content":"You are ChatGPT, a large language model trained by OpenAI.\nCarefully heed the user's instructions. \nRespond using Markdown."
      },
      {
         "role":"user",
         "content":"hi"
      }
   ],
   "model":"@cf/meta/llama-2-7b-chat-int8",
   "temperature":1,
   "presence_penalty":0,
   "top_p":1,
   "frequency_penalty":0,
   "stream":true
}

Is there anything I should remove/add from/to the request body?

[chand1012]

No that was my bad again, fixed with the latest commit. If you're curious as to what I screwed up it was this: 6cb8425.

[chitudorin]

It works fine now, I get a response, but BetterChatGPT currently fails to display it.

Response:

{
   "id":"00b07baa-e30e-4990-8f5c-6f9beb144a55",
   "model":"@hf/thebloke/llama-2-13b-chat-awq",
   "created":1711469791,
   "object":"chat.completion",
   "choices":[
      {
         "index":0,
         "message":{
            "role":"assistant",
            "content":"  Hello! 👋 I'm here to help answer any questions you may have. Please keep in mind that I'm just an AI and do not have personal experiences or emotions. I will do my best to provide helpful and accurate information to the best of my ability based on my training. What would you like to know or discuss? 🤔"
         },
         "finish_reason":"stop"
      }
   ],
   "usage":{
      "prompt_tokens":0,
      "completion_tokens":0,
      "total_tokens":0
   }
}

I'll see if I can fix it from BetterChatGPT's source or open an issue on their repo. Feel free to close this.

[chitudorin]

I think BetterChatGPT only accepts streaming responses so since Cloudflare's isn't it's not processing it at all.

[chand1012]

Streaming responses are on my list of todos but I have no ETA.

…

On Tue, Mar 26, 2024 at 1:21 PM Chitu Dorin ***@***.***> wrote: I think BetterChatGPT only accepts streaming responses <ztjhz/BetterChatGPT#290 (comment)> so since Cloudflare's isn't it's not processing it at all. — Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA23YLRUL6NDV3LD4UJZ6XLY2GN7VAVCNFSM6AAAAABFGSHOBGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRRGA2DCMZWGQ> . You are receiving this because you were assigned.Message ID: ***@***.***>

[chand1012]

@chitudorin quick update on this, as of PR #5, streaming is now supported on the main branch.

[chitudorin]

@chand1012 I randomly checked the repo like 10 hours ago and saw the PR. Tested it then and confirmed it works, nice!