Erratic behavior of useChat - Could SWR be the issue?

[martinjo]

Description

I've been having some trouble getting consistent behavior from this library when using useChat in my Svelte application. Updating, reloading, and deleting messages have been erratic, and I'm not sure what I'm doing wrong.

After trying everything I could think of in my application, I decided to replace SWR in useChat with a regular Svelte store to keep track of the chats. This simple change made the library behave exactly as I expected. Now, I can call useChat at any point in the application, not just during component initialization as it seemed to require before. Since making this change, I haven't experienced any issues.

What is the advantage of using SWR for Svelte users?

Also, I tried to install the library from a Github fork, but I encountered an error saying that the name is missing. Is it possible to install directly from a Github fork? How can I do that? To test my changes I had to incorporate the source into my project and naturally this will be hard to maintain...

Code example

No response

Additional context

No response

[lgrammel]

You can edit the examples inside the fork pretty easily (using pnpm build on root and pnpm build in packages/core). You can use linking mechanism for projects outside.

[martinjo]

I managed to build the examples and test my changes there. But I had no luck in linking the library with my changes (only svelte/use-chat.ts was changed) into my project. Can you be more specific in how to do this?

Also have you experienced any of the problems I've seen with useChat in svelte?

Below is my use-chat.ts if someone else is having the same problems as I've had:

import { type Readable, type Writable, get, writable } from 'svelte/store';
import { callChatApi } from '../shared/call-chat-api';
import { processChatStream } from '../shared/process-chat-stream';
import type {
  ChatRequest,
  ChatRequestOptions,
  CreateMessage,
  IdGenerator,
  JSONValue,
  Message,
  UseChatOptions,
} from 'ai';
import { generateId as generateIdFunc } from '../shared/generate-id';
export type { CreateMessage, Message, UseChatOptions };

export type UseChatHelpers = {
  messages: Readable<Message[]>;
  error: Readable<undefined | Error>;
  append: (
    message: Message | CreateMessage,
    chatRequestOptions?: ChatRequestOptions,
  ) => Promise<string | null | undefined>;
  reload: (
    chatRequestOptions?: ChatRequestOptions,
  ) => Promise<string | null | undefined>;
  stop: () => void;
  setMessages: (messages: Message[]) => void;
  input: Writable<string>;
  handleSubmit: (e: any, chatRequestOptions?: ChatRequestOptions) => void;
  metadata?: Object;
  isLoading: Readable<boolean | undefined>;
  data: Readable<JSONValue[] | undefined>;
};

const getStreamedResponse = async (
  api: string,
  chatRequest: ChatRequest,
  mutate: (messages: Message[]) => void,
  mutateStreamData: (data: JSONValue[] | undefined) => void,
  existingData: JSONValue[] | undefined,
  extraMetadata: {
    credentials?: RequestCredentials;
    headers?: Record<string, string> | Headers;
    body?: any;
  },
  previousMessages: Message[],
  abortControllerRef: AbortController | null,
  generateId: IdGenerator,
  streamMode?: 'stream-data' | 'text',
  onFinish?: (message: Message) => void,
  onResponse?: (response: Response) => void | Promise<void>,
  sendExtraMessageFields?: boolean,
) => {
  mutate(chatRequest.messages);

  const constructedMessagesPayload = sendExtraMessageFields
    ? chatRequest.messages
    : chatRequest.messages.map(
        ({ role, content, name, function_call, tool_calls, tool_call_id }) => ({
          role,
          content,
          tool_call_id,
          ...(name !== undefined && { name }),
          ...(function_call !== undefined && {
            function_call: function_call,
          }),
          ...(tool_calls !== undefined && {
            tool_calls: tool_calls,
          }),
        }),
      );

  return await callChatApi({
    api,
    messages: constructedMessagesPayload,
    body: {
      ...extraMetadata.body,
      ...chatRequest.options?.body,
      ...(chatRequest.functions !== undefined && {
        functions: chatRequest.functions,
      }),
      ...(chatRequest.function_call !== undefined && {
        function_call: chatRequest.function_call,
      }),
      ...(chatRequest.tools !== undefined && {
        tools: chatRequest.tools,
      }),
      ...(chatRequest.tool_choice !== undefined && {
        tool_choice: chatRequest.tool_choice,
      }),
    },
    streamMode,
    credentials: extraMetadata.credentials,
    headers: {
      ...extraMetadata.headers,
      ...chatRequest.options?.headers,
    },
    abortController: () => abortControllerRef,
    restoreMessagesOnFailure() {
      mutate(previousMessages);
    },
    onResponse,
    onUpdate(merged, data) {
      mutate([...chatRequest.messages, ...merged]);
      mutateStreamData([...(existingData || []), ...(data || [])]);
    },
    onFinish,
    generateId,
  });
};

const chatStores = new Map<string, UseChatHelpers>();

export function useChat({
  api = '/api/chat',
  id,
  initialMessages = [],
  initialInput = '',
  sendExtraMessageFields,
  experimental_onFunctionCall,
  experimental_onToolCall,
  streamMode,
  onResponse,
  onFinish,
  onError,
  credentials,
  headers,
  body,
  generateId = generateIdFunc,
}: UseChatOptions = {}): UseChatHelpers {
  const chatId = id || `chat-${generateId()}`;

  if (chatStores.has(chatId)) {
    return chatStores.get(chatId) as UseChatHelpers;
  }

  const messages = writable<Message[]>(initialMessages);
  const streamData = writable<JSONValue[] | undefined>(undefined);
  const isLoading = writable<boolean>(false);
  const error = writable<undefined | Error>(undefined);
  const input = writable(initialInput);

  const mutate = (data: Message[]) => {
    messages.set(data);
  };

  let abortController: AbortController | null = null;

  const extraMetadata = {
    credentials,
    headers,
    body,
  };

  async function triggerRequest(chatRequest: ChatRequest) {
    try {
      error.set(undefined);
      isLoading.set(true);
      abortController = new AbortController();

      await processChatStream({
        getStreamedResponse: () =>
          getStreamedResponse(
            api,
            chatRequest,
            mutate,
            data => {
              streamData.set(data);
            },
            get(streamData),
            extraMetadata,
            get(messages),
            abortController,
            generateId,
            streamMode,
            onFinish,
            onResponse,
            sendExtraMessageFields,
          ),
        experimental_onFunctionCall,
        experimental_onToolCall,
        updateChatRequest: chatRequestParam => {
          chatRequest = chatRequestParam;
        },
        getCurrentMessages: () => get(messages),
      });

      abortController = null;

      return null;
    } catch (err) {
      if ((err as any).name === 'AbortError') {
        abortController = null;
        return null;
      }

      if (onError && err instanceof Error) {
        onError(err);
      }

      error.set(err as Error);
    } finally {
      isLoading.set(false);
    }
  }

  const append: UseChatHelpers['append'] = async (
    message: Message | CreateMessage,
    {
      options,
      functions,
      function_call,
      tools,
      tool_choice,
    }: ChatRequestOptions = {},
  ) => {
    if (!message.id) {
      message.id = generateId();
    }

    const chatRequest: ChatRequest = {
      messages: get(messages).concat(message as Message),
      options,
      ...(functions !== undefined && { functions }),
      ...(function_call !== undefined && { function_call }),
      ...(tools !== undefined && { tools }),
      ...(tool_choice !== undefined && { tool_choice }),
    };
    return triggerRequest(chatRequest);
  };

  const reload: UseChatHelpers['reload'] = async ({
    options,
    functions,
    function_call,
    tools,
    tool_choice,
  }: ChatRequestOptions = {}) => {
    const messagesSnapshot = get(messages);
    if (messagesSnapshot.length === 0) return null;

    const lastMessage = messagesSnapshot.at(-1);
    if (lastMessage?.role === 'assistant') {
      const chatRequest: ChatRequest = {
        messages: messagesSnapshot.slice(0, -1),
        options,
        ...(functions !== undefined && { functions }),
        ...(function_call !== undefined && { function_call }),
        ...(tools !== undefined && { tools }),
        ...(tool_choice !== undefined && { tool_choice }),
      };

      return triggerRequest(chatRequest);
    }
    const chatRequest: ChatRequest = {
      messages: messagesSnapshot,
      options,
      ...(functions !== undefined && { functions }),
      ...(function_call !== undefined && { function_call }),
      ...(tools !== undefined && { tools }),
      ...(tool_choice !== undefined && { tool_choice }),
    };

    return triggerRequest(chatRequest);
  };

  const stop = () => {
    if (abortController) {
      abortController.abort();
      abortController = null;
    }
  };

  const setMessages = (messages: Message[]) => {
    mutate(messages);
  };

  const handleSubmit = (e: any, options: ChatRequestOptions = {}) => {
    e.preventDefault();
    const inputValue = get(input);
    if (!inputValue) return;

    append(
      {
        content: inputValue,
        role: 'user',
        createdAt: new Date(),
      },
      options,
    );
    input.set('');
  };

  const chatHelpers: UseChatHelpers = {
    messages,
    error,
    append,
    reload,
    stop,
    setMessages,
    input,
    handleSubmit,
    isLoading,
    data: streamData,
  };

  chatStores.set(chatId, chatHelpers);

  return chatHelpers;
}

[daveycodez]

The new updates to the AI SDK are doing some crazy things with SWR and it is malfunctioning with my custom SWR Config... Might not be able to use the useChat hook at all and stick to API routes only here.

[lgrammel]

swr has been removed from useChat in ai sdk 5