run / wait can get stuck when using Next.js with App Router

[mattt]

At yesterday's hackathon, a participant shared that predictions created and polled for completion using run would hang indefinitely, despite the Replicate dashboard showing that prediction as finishing minutes earlier.

After debugging by adding a console.log statement in the progress callback function, we determined that this behavior was caused by the extensions to fetch made by Next.js when using App Router ¹. From "Data Fetching, Caching, and Revalidating" in the Next.js docs:

Next.js extends the native fetch Web API to allow you to configure the caching and revalidating behavior for each fetch request on the server. React extends fetch to automatically memoize fetch requests while rendering a React component tree.
...
By default, Next.js automatically caches the returned values of fetch in the Data Cache on the server. This means that the data can be fetched at build time or request time, cached, and reused on each data request.

Because the initial GET /v1/predictions/{id} response was cached, subsequent requests polling for status returned the same response, which had the initial "starting" status.

Our workaround involved setting replicate.fetch to a wrapped function that specified a cache: no-store fetch option. But it's unclear whether there's a better way to get the desired behavior.

We should either document or add a workaround to make this work correctly in Next.js.

Footnotes

1. fetch when using Pages Router, like in the "Build a website with Next.js" sample project, works as expected. ↩

[mirko314]

Hey,
I also just ran into this, and it took me quite some time to debug and come to the conclusion that some "magical" caching must be the culprit.

Would you mind to share your workaround code?

With App Router and Vercel getting even more traction, this definatly should be addressed!

Thanks!

[mattt]

@mirko314 Thanks for sharing. That's a helpful data point.

Here's what I remember the code to be. Please give this a try and let me know if that works for you:

replicate = new Replicate({/*...*/})
replicate.fetch = (url, options) => {
  return fetch(url, { ...options, cache: "no-store" });
};

[mirko314]

Thanks @mattt, that worked well!

[Autometrique]

@mattt @mirko314
Hi!
Thanks for this very helpful comment.
I ran into this too. I'm fairly new to next.js with the app router and having a hard time trying to make it work.
Maybe would you have a more complete exemple to share, showing how you managed to make it work?

I'm trying to use it within a replicate create/wait structure (which is equivalent to run/wait)

That would be fantastic :-)

I agree this should be addressed. We won't be the only ones trying to use replicate on the new next.js app router.

Thanks!

[pondorasti]

Just ran into the same issue and this #136 (comment) fixed it for me too. Thanks for the post @mattt, probably saved me hours of troubleshooting 🙌

[Autometrique]

@mattt Would it be possible to update the documentation with a more detailed explanation on how to use replicate-javascript on next.js recent versions? 😃

[leerob]

Rather than changing the Replicate fetch behavior, you can use noStore to opt-out of caching for your component.

import Replicate from 'replicate';
import { unstable_noStore as noStore } from 'next/cache';

let replicate = new Replicate(...);
 
export default async function Component() {
  noStore();
  let prediction = await replicate.predictions.create(...)
  ...
}

API: https://nextjs.org/docs/app/api-reference/functions/unstable_noStore

[zeke]

I've opened a couple PRs to help prevent further confusion about this:

• Update README with troubleshooting section #264
• Prevent response caching in Next.js getting-started-nextjs#104

[wuyasong]

您无需改变复制fetch行为，就可以选择noStore退出组件的缓存。

import Replicate from 'replicate';
import { unstable_noStore as noStore } from 'next/cache';

let replicate = new Replicate(...);
 
export default async function Component() {
  noStore();
  let prediction = await replicate.predictions.create(...)
  ...
}

API：https：//nextjs.org/docs/app/api-reference/functions/unstable_noStore

This Great, I was stuck with this issue for a day, and after reading this issue, it was finally fixed