Support client-only use of chat library

[jeloi]

There are applications that may not want to proxy all LLM calls through a backend server, which is a current limitation.

Specifically, the useChat hook in React assumes making a fetch call to the server, with a specific response type.

One option we've attempted is to pass in a remote url for the model provider's api directly into the useChat config - however this falls short because the decoding step is still missed (as that is assumed to happen on the server).

It would be very nice if the same abstractions around models could be used on the client - e.g passing in a config to produce a given stream.

Thank you for building this!

[rozenstein94]

its against the idea of project, and in general you need backend not to leak API keys. Adds extra layer of complexity as well as we will need local storage/login capabilities

[MaxLeiter]

Thanks for the request @jeloi, but for the reasons @rozenstein94 shared I'm going to close this as out-of-scope.

[marjoweb]

I understand it not the intended intended use of the library, but wouldn’t client side only functionality be a good fit to use with a local ollama instance? How do you recommend leveraging that with the libary now?

[lgrammel]

There are use cases for being able to use this without fetch, e.g. client side usage with client AI models or usage in RSCs.

For now, you can pass in a custom fetch function to replace the fetch behavior with some mock/client-only implementation.

[sethhorsley]

I agree 100%. Another big use case like a client side app where each user uses his own api keys and it would be stored locally and not need any server to execute the code.

[Glavin001]

I'd love to contribute a Pull Request for this! I could use help though.

Is there a working user-land workaround example I could try to make a first-class supported feature?

For now, you can pass in a custom fetch function to replace the fetch behavior with some mock/client-only implementation.

&

... pass in a remote url for the model provider's api directly into the useChat config - however this falls short because the decoding step is still missed (as that is assumed to happen on the server).

@jeloi & @lgrammel & any others: Could you provide some example code snippets you've tried and what works and does not work with them? Sounds like a custom fetch function parameter is step in the right direction, however, still misses functionality in a server-side decoding step. What exactly is missed?

[sethhorsley]

I have this code working without any backend api:

'use client'

import { useState, useCallback, useRef, useEffect } from 'react'
import { useChat, Message } from 'ai/react'

export default function ChatUI() {
  const [streaming, setStreaming] = useState(false)
  const { messages, setMessages, input, handleInputChange, handleSubmit } = useChat({
    api: '/api/chat',
    onFinish: () => setStreaming(false),
  })

  const abortControllerRef = useRef<AbortController | null>(null)

  const streamOpenAIResponse = useCallback(async (userMessage: string) => {
    console.log('Starting streamOpenAIResponse with message:', userMessage)
    setStreaming(true)
    const newMessage: Message = { id: `${Date.now()}-${Math.random().toString(36).substr(2, 9)}`, content: '', role: 'assistant' }
    setMessages((prevMessages) => [...prevMessages, newMessage])

    abortControllerRef.current = new AbortController()

    try {
      console.log('Making fetch request to OpenAI...')
      console.log('API Key available:', !!process.env.NEXT_PUBLIC_OPENAI_API_KEY)
      
      const requestBody = {
        model: 'gpt-3.5-turbo',
        messages: [{ role: 'user', content: userMessage }],
        stream: true,
      }
      console.log('Request body:', requestBody)

      const response = await fetch('https://api.openai.com/v1/chat/completions', {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          'Authorization': `Bearer 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`,
        },
        body: JSON.stringify(requestBody),
        signal: abortControllerRef.current.signal,
      })

      console.log('Response status:', response.status)
      console.log('Response OK:', response.ok)

      if (!response.ok) {
        const errorData = await response.text()
        console.error('Error response:', errorData)
        throw new Error(`HTTP error! status: ${response.status}, message: ${errorData}`)
      }

      const reader = response.body?.getReader()
      if (!reader) {
        throw new Error('Response body reader is null')
      }
      
      console.log('Starting to read stream...')
      const decoder = new TextDecoder('utf-8')

      while (true) {
        const { done, value } = await reader.read()
        if (done) {
          console.log('Stream complete')
          setStreaming(false)
          break
        }

        // Decode the stream data
        const chunk = decoder.decode(value)
        console.log('Received chunk:', chunk)
        
        const lines = chunk.split('\n').filter(line => line.trim() !== '')
        console.log('Processed lines:', lines)

        for (const line of lines) {
          if (line.includes('[DONE]')) {
            console.log('Received [DONE] signal')
            continue
          }
          if (line.startsWith('data: ')) {
            try {
              const data = JSON.parse(line.slice(6))
              console.log('Parsed data:', data)
              
              if (data.choices[0]?.delta?.content) {
                const content = data.choices[0].delta.content
                console.log('New content:', content)
                setMessages((prevMessages) =>
                  prevMessages.map((msg) =>
                    msg.id === newMessage.id
                      ? { ...msg, content: msg.content + content }
                      : msg
                  )
                )
              }
            } catch (e) {
              console.error('Error parsing JSON:', e)
              console.error('Problematic line:', line)
            }
          }
        }
      }
    } catch (error) {
      console.error('Caught error:', error)
      console.error('Error name:', error.name)
      console.error('Error message:', error.message)
      console.error('Error stack:', error.stack)
      
      if (error.name === 'AbortError') {
        console.log('Fetch aborted')
      } else {
        console.error('Error:', error)
      }
    } finally {
      console.log('Cleaning up...')
      setStreaming(false)
    }
  }, [setMessages])

  const handleFormSubmit = useCallback((e: React.FormEvent<HTMLFormElement>) => {
    console.log('Form submitted')
    e.preventDefault()
    if (streaming) {
      console.log('Aborting current stream')
      abortControllerRef.current?.abort()
      setStreaming(false)
      return
    }
    if (input.trim()) {
      console.log('Processing input:', input)
      const userMessage: Message = { id: `${Date.now()}-${Math.random().toString(36).substr(2, 9)}`, content: input, role: 'user' }
      setMessages((prevMessages) => [...prevMessages, userMessage])
      handleInputChange({ target: { value: '' } } as React.ChangeEvent<HTMLInputElement>)
      streamOpenAIResponse(input)
    }
  }, [input, streaming, handleInputChange, setMessages, streamOpenAIResponse])

  useEffect(() => {
    return () => {
      console.log('Component cleanup - aborting any ongoing requests')
      abortControllerRef.current?.abort()
    }
  }, [])

  // Rest of your component remains the same...
  return (
    <div className="flex flex-col h-screen max-w-2xl mx-auto p-4">
      <div className="flex-1 overflow-y-auto mb-4">
        {messages.map((m, index) => (
          <div key={`${m.id}-${index}`} className={`mb-4 ${m.role === 'user' ? 'text-right' : 'text-left'}`}>
            <span className={`inline-block p-2 rounded-lg ${m.role === 'user' ? 'bg-blue-500 text-white' : 'bg-gray-200 text-black'}`}>
              {m.content}
            </span>
          </div>
        ))}
      </div>
      <form onSubmit={handleFormSubmit} className="flex space-x-2">
        <input
          className="flex-1 p-2 border rounded"
          value={input}
          onChange={handleInputChange}
          placeholder="Type your message..."
        />
        <button type="submit" className="p-2 bg-blue-500 text-white rounded">
          {streaming ? 'Stop' : 'Send'}
        </button>
      </form>
    </div>
  )
}

[iworkinprogress]

We have a separate backend that is not through Next and adheres to an Open AI schema. Would love to be able to point useChat to our existing endpoints and have them work without having to proxy everything through a route in our Next.js app.

[SerekKiri]

@sethhorsley hopefully you rotated the key from the completions call ;)

[lgrammel]

ai sdk 5 introduces the option to use custom chat transports, which enables client only usage.