Edge functions streaming response with long timeout

[drorIvry]

Summary

Hi, I'm trying to run an edge function that has a long (45+) seconds wait time (in which no data is being streamed)

for some reason the stream is cut after 35 seconds and the response won't be streamed
Here's a simplified example

Given this route
/pages/api/test.ts

export const config = {
  runtime: "edge",
};

function sleep(ms: number) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

export default function handler() {
  const encoder = new TextEncoder();

  const customReadable = new ReadableStream({
    async start(controller) {

      await sleep(1000 * 45); // big block of synchronous calculations

      controller.enqueue(encoder.encode("end"));
      setTimeout(() => controller.close(), 300);
    },
  });

  return new Response(customReadable, {
    headers: { "Content-Type": "text/html; charset=utf-8" },
  });
}

curl  -X GET \
  'https://******/api/test/' \
  --header 'Accept: */*' \
  --header 'User-Agent: Thunder Client (https://www.thunderclient.com)'

the request will hang for 35 seconds and simply finish with status 200, any idea what I'm missing here?
Is there a better way to do what I'm trying to do?

Example

No response

Steps to Reproduce

1. create a clean app with the example route
2. deploy to vercel
3. run the cURL command

[leerob]

I see you listed pages/ – is this a Next.js application?

If so, you need to use Route Handlers. API Routes (from the Pages Router) do not support streaming responses on Vercel.

[drorIvry]

Hi @leerob I migrated to the appdir and created a route handler as instructed

app/api/test/route.ts

export const runtime = "edge";

function sleep(ms: number) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

export async function GET() {
  const encoder = new TextEncoder();

  const customReadable = new ReadableStream({
    async start(controller) {
      await sleep(1000 * 45); // big block of synchronous calculations

      controller.enqueue(encoder.encode("end"));
      setTimeout(() => controller.close(), 300);
    },
  });
  await sleep(1000);
  return new Response(customReadable, {
    headers: { "Content-Type": "text/html; charset=utf-8" },
  });
}

and it's the same issue as the one with the pages router. In local when calling /api/test the route hangs for 45 seconds and get the end response. when deployed to vercel though, the request hangs for 35 seconds and terminates without the end message.

I ran a new test where I continuously stream data for 45 seconds and that works which tells me that vercel kills the edge worker after 35 seconds of no data being streams. As in my case it's a sync block of code in the middle of the stream I can't reduce the time with no data being streamed. Is there a "keep alive" option I'm missing? do you have any suggestion to solve this?

thanks!

[leerob]

The route needs to start producing a response in the first 30 seconds, then can continue streaming the response for as long as you need.

[drorIvry]

@leerob thanks!
I also tried this:

export const runtime = "edge";

function sleep(ms: number) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

export async function GET() {
  const encoder = new TextEncoder();

  const customReadable = new ReadableStream({
    async start(controller) {
      controller.enqueue(encoder.encode("satrt"));

      await sleep(1000 * 45); // big block of synchronous calculations

      controller.enqueue(encoder.encode("end"));
      setTimeout(() => controller.close(), 300);
    },
  });
  await sleep(1000);
  return new Response(customReadable, {
    headers: { "Content-Type": "text/html; charset=utf-8" },
  });
}

In local when calling /api/test the route hangs for 45 seconds and get the start and end response. When deployed to Vercel though, the request hangs for 35 seconds and terminates without the end message only outputting "start".

[BenSchZA]

@drorIvry I can confirm I'm having the same issue. When streaming, if there's an initial response and then a gap, it will timeout after 35 seconds. If there's no initial response it will timeout after the documented 60 seconds for the Pro plan. I resorted to sending a keep alive event every 15 seconds which resolves the issue, but sometimes it's not possible to keep sending these keep alive events that regularly (for example if there's a CPU-bound job that blocks for longer than 35 seconds).

Did you find another solution?

[drorIvry]

@BenSchZA sadly couldn't find a solution with the vercel runtime, ended up moving that service off of vercel

[BenSchZA]

@drorIvry I can confirm that streaming a periodic "keep alive" message every 15 seconds ended up resolving the (seemingly 35 second) timeout:

export async function POST(req: Request): Promise<Response> {
  const json = await req.json()
  ...
  const userId = (await auth())?.user.id

  if (!userId) {
    return new Response('Unauthorized', {
      status: 401
    })
  }

  try {
    const response = await fetch(`.../stream`, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({...})
    })

    if (!response.body || !response.ok) {
      console.error(response)
      return new Response('API error', {
        status: 500
      })
    }

    const encoder = new TextEncoder()
    const decoder = new TextDecoder()

    let completion = ''

    const stream = new ReadableStream({
      async start(controller) {
        let streamEnded = false;

        async function keepAlive() {
          while (!streamEnded) {
            const space = '\u200B'; // Zero Width Space
            const queue = encoder.encode(space);
            controller.enqueue(queue);
            await new Promise(resolve => setTimeout(resolve, 15_000));
          }
        }

        async function onParse(event: ParsedEvent | ReconnectInterval): Promise<void> {
          if (event.type === "event") {
            const data = event.data;
            if (event.event === "stream_start"){
              console.log(`Stream has started`)
              const output_value = '...';
              const queue = encoder.encode(output_value);
              completion += output_value;
              controller.enqueue(queue);
            } else if (event.event === "stream_end" || data === "[DONE]"){
              console.log(`Stream has ended`)
              streamEnded = true;
              const output_value = '...';
              const queue = encoder.encode(output_value);
              completion += output_value;
              controller.enqueue(queue);

              await onCompletion(...);
              controller.close();
              return;
            } else {
              ...
            }
          }
        }

        keepAlive();

        // stream response (SSE) may be fragmented into multiple chunks
        // this ensures we properly read chunks & invoke an event for each SSE event stream
        const parser = createParser(onParse);

        // https://web.dev/streams/#asynchronous-iteration
        for await (const chunk of response.body as any) {
          parser.feed(decoder.decode(chunk));
        }
      },
    });

    return new StreamingTextResponse(stream)
  } catch (error) {
    console.error(error)
    return new Response('API error', {
      status: 500,
    })
  }
}

It seems that the Vercel docs aren't clear on this matter:

Edge Functions must begin sending a response within 25 seconds. You may continue streaming a response beyond that time for as long as needed. Learn more about the supported runtime APIs and limits.

Specifically, apart from sending a response within 25 seconds (which I currently do) you actually have to keep sending messages periodically (unclear at what frequency, but 15 seconds resolves the issue) to keep the stream alive.

You may continue streaming a response beyond that time for as long as needed.

I've tested, and you can't enqueue empty messages either - the utf-8 encoded value has to contain non-zero bytes, which is why I used a "Zero Width Space" so it doesn't appear on the client side. Instead, it appears as <p class="mb-2 last:mb-0">​​​&ZeroWidthSpace;&ZeroWidthSpace;&ZeroWidthSpace;</p> which seems like a bit of a hack :D

[khoa-lucents]

Thanks for the workaround