Example with Next 13 / Pages Router / NextAuth 4

[scvsh]

This is how you can run streaming in the old pages/api route. Surely it's subject to many improvements, but the crucial part — streaming and authentication work correctly without switching to app router.

import { LangChainStream, Message } from "ai";
import { ChatOpenAI } from "langchain/chat_models/openai";
import { AIChatMessage, HumanChatMessage } from "langchain/schema";
import { authOptions } from "@/server/auth";
import { getServerSession } from "next-auth/next";
import { NextApiRequest, NextApiResponse } from "next";
import EventEmitter from "events";

// This is a workaround to generate events from an LLM ReadableStream
class ReadableStreamEmitter extends EventEmitter {
  constructor(readableStream: ReadableStream) {
    super();
    this.startReading(readableStream);
  }

  async startReading(readableStream: ReadableStream) {
    const reader = readableStream.getReader();

    while (true) {
      const { done, value } = await reader.read();

      if (done) {
        this.emit("end");
        break;
      }

      this.emit("data", value);
    }

    reader.releaseLock();
  }
}

async function handler(req: NextApiRequest, res: NextApiResponse<any>) {
  // Get the user's session based on the request, works with node runtime / Next Auth 4
  const session = await getServerSession(req, res, authOptions);
  const { messages } = JSON.parse(req.body);

  const { stream, handlers } = LangChainStream();
  const eventStream = new EventEmitter();
  const streamEmitter = new ReadableStreamEmitter(stream);

  const llm = new ChatOpenAI({
    openAIApiKey: process.env.OPENAI_API_KEY,
    streaming: true,
    callbacks: [handlers],
  });

  llm
    .call(
      (messages as Message[]).map((m) =>
        m.role == "user"
          ? new HumanChatMessage(m.content)
          : new AIChatMessage(m.content)
      )
    )
    .catch(console.error);

  // Write headers for event-stream connection
  res.writeHead(200, {
    Connection: "keep-alive",
    "Content-Encoding": "none",
    "Cache-Control": "no-cache, no-transform",
    "Content-Type": "text/event-stream",
  });

  // Create a new server-sent event stream
  eventStream.on("channel", function (event, data) {
    res.write(data);
  });

  // When the LLM emits data, send it to the client
  streamEmitter.on("data", (chunk) => {
    const resp = new TextDecoder().decode(chunk);
    eventStream.emit("channel", "message", resp); // the event name here must be the same as in the EventSource in frontend
  });

  // If langChain stream ends, end the response
  streamEmitter.on("end", () => {
    res.end("done\n");
  });

  // If the client closes the connection, end the response
  res.socket?.on("close", () => {
    res.end();
  });
}

export default handler;

export const config = {
  api: {
    externalResolver: true,
  },
};

[MaxLeiter]

I’ll work next week to update the documentation, but you can use the generate code button on the https://sdk.vercel.ai playground to generate app router, pages router, svelte and node.js code

[scvsh]

@MaxLeiter Thank you for the attention to this!

I'm using node.js runtime with pages router. The sdk playground has an example for this stack, but it doesn't seem to work. The problem I get with the sdk generated example is that it returns a StreamingTextResponse from the route handler, which results in API route returned a Response object in the Node.js runtime, this is not supported error. As a solution it suggests switching to edge runtime. This switch is not working for me unfortunately, since in that case NextAuth session will not be available in the api route.

This led me to take the approach above and manually process streaming messages on the server side, which I consider suboptimal. If there is another way to return the Response object from nodejs api I would be happy to adopt!

[MaxLeiter]

Hmm, Next Auth should be compatible with the Edge runtime. @scvsh would you mind opening a repository with a reproduction I can test and share with the relevant parties? Thank you!

[scvsh]

As per my understanding only the latest pre-release NextAuth version (Auth.js) works correctly with Edge. I made a testing environment, it uses pages router, NextAuth v4 and sdk example with edge runtime, falling with TypeError: Cannot read properties of undefined (reading 'substring') upon api request execution