Workflow scaling issues

[vahagn-madatyan]

Hi all,

We have been testing a deployment in our application that pushes the scale of the workflow system by running a deployment that would produce 100x iterations for group of steps(~5 steps).

• Each iteration is executed sequentially, all steps are external API calls.
• So we have 1 workflow with 5 steps that execute external API calls. One of the steps long running for ~30 minutes. 5 steps are in a loop based on user input of qty.
• So we executed 100x > 1x workflow (5 steps * 100), 500 steps in Vercel World. Run has been going for 2+ hours.

This produced 1000 events in workflow as its going through its iteration. Once it reached 1000 events in workflow, it completly broke and hang up. We are seeing the following errors in Vercel logs. The run is in progress state but only produces these errors.

Queue callback error: Error [WorkflowAPIError]: workflow step step_01KH7ATNEGDTGCRZP97F42ZYEZ not found at ignore-listed frames { cause: undefined, status: 404, code: undefined, url: 'https://vercel-workflow.com/api/v1/runs/wrun_01KH7ATFXVM4Z6WWZG57EYJXN7/steps/step_01KH7ATNEGDTGCRZP97F42ZYEZ?remoteRefBehavior=resolve' }

and

Error: Failed to find Server Action "009b8da430e8bc0ffef7dfbbe7a31524830b1a36a2". This request might be from an older or newer deployment. Read more: https://nextjs.org/docs/messages/failed-to-find-server-action at ignore-listed frames

We also noticed increasing number of Vercel Runtime Timeout Error: Task timed out after 800 seconds, which does not affect the workflow but causes stream to hang up sometimes to UI communicating progress updates.

This is real scenario where we would run this in production every day. What is the recommended approach to implement Vercel workflows for such large run( 5 steps of external api calls, long running, that need ot be done in groups of 100-200s daily)?

[pranaygp]

hey @vahagn-madatyan - from that run you shared, it looks like you're on an older version of workflow. I'd recommend upgrading to 4.1.0-beta where we've addressed a number of stability issues and we shouldn't see the step not found error there. Could you please let mw know if you still do.

will get back to you on the other issues you mentioned soon. one thing on streams in the meantime is to make sure you're closing your streams when you're done. Keeping them open won't affect the workflow run or your execution/cost but will unnecessarily keep flow routes running until the function eventually times out

[vahagn-madatyan]

hey @vahagn-madatyan - from that run you shared, it looks like you're on an older version of workflow. I'd recommend upgrading to 4.1.0-beta where we've addressed a number of stability issues and we shouldn't see the step not found error there. Could you please let mw know if you still do.

will get back to you on the other issues you mentioned soon. one thing on streams in the meantime is to make sure you're closing your streams when you're done. Keeping them open won't affect the workflow run or your execution/cost but will unnecessarily keep flow routes running until the function eventually times out

Thanks will update and check. If I close the stream, the client will stop receiving the progress updates. In our client side, we have like a live progress bar. How can we keep that live for long duration acorss sleeps & function timeouts? We are enterprise customer, so timeout is at 800 max. For now we have implemented "Resume stream button" but thats not user friendly for people who are monitoring progress on the side.

[VaguelySerious]

we have implemented "Resume stream button"

@vahagn-madatyan Is there a reason that auto-reconnecting on stream close would not work for this case?

[vahagn-madatyan]

we have implemented "Resume stream button"

@vahagn-madatyan Is there a reason that auto-reconnecting on stream close would not work for this case?

If we can auto recconnect silently that would work, I tried to see how i can detect disconnect event to resume but was not able to find a way. Is there an example on this? I saw the other issue linked to how Durable Agent does it with useChat, but we are not using that.

[vahagn-madatyan]

hey @vahagn-madatyan - from that run you shared, it looks like you're on an older version of workflow. I'd recommend upgrading to 4.1.0-beta where we've addressed a number of stability issues and we shouldn't see the step not found error there. Could you please let mw know if you still do.

will get back to you on the other issues you mentioned soon. one thing on streams in the meantime is to make sure you're closing your streams when you're done. Keeping them open won't affect the workflow run or your execution/cost but will unnecessarily keep flow routes running until the function eventually times out

The version upgrade appears to have worked, its going further than it did yesterday, I also noticed you guys pushed updates to event view in Vercel workflows to lazy load more than 1000 events.

Few remaining items are:

1. The stream continue auto-reconnect solution
2. The timeflow of the workflow bugs out for large events and stops showing things:

3. We got these error in the log thats new

Queue callback error: Error [WorkflowAPIError]: workflow step step_01KH7JJ64GJPFZMDZ8MQ228H7N not found at ignore-listed frames { cause: undefined, status: 404, code: undefined, url: 'https://vercel-workflow.com/api/v1/runs/wrun_01KH7JJ4T28WQBRVWA81175GWP/steps/step_01KH7JJ64GJPFZMDZ8MQ228H7N?remoteRefBehavior=resolve' }

and

Failed to extend visibility for receipt handle s.3.1M0uwkt23L3HOHpo0pvrYNJpfrGzq0gM.GhrU8eOE4y2: Error [MessageNotAvailableError]: Message s.3.1M0uwkt23L3HOHpo0pvrYNJpfrGzq0gM.GhrU8eOE4y2 not available for processing: {"error":"Message is not currently in-flight"} at ignore-listed frames

[pranaygp]

@vahagn-madatyan ah sorry I didn't mean to say "closing your stream". I meant to say you should release the lock. that's my bad. in a step, you can release the lock on the stream after you're done writing to it. that should be enough to let the function end properly without timing out, but the stream won't be closed and other steps in the workflow can continue writing to it. The stream will automatically get closed at the end of the run (or you can always have a final step that closes it manually too)

[pranaygp]

Clients that are consiming the stream will still need to reconnect if you go past the 15 minute timeout, but that retrying is pretty straightforward. There's a code example here (https://github.com/vercel/workflow-examples/tree/main/flight-booking-app). You can see how we return a stream (in this case, it's using DurableAgent, but any stream works. On the client, it uses WorkflowChatTransport to resume the stream across refreshes and disconnects. The pattern is pretty simple. You can even point claude to look at the flight booking example source code, or even point it to source code of @workflow/ai package here to see how the resumption works client side

[pranaygp]

The timeflow of the workflow bugs out for large events and stops showing things:
We got these error in the log thats new ...

We'll dig into those asap - thanks for sharing

[vahagn-madatyan]

Clients that are consiming the stream will still need to reconnect if you go past the 15 minute timeout, but that retrying is pretty straightforward. There's a code example here (https://github.com/vercel/workflow-examples/tree/main/flight-booking-app). You can see how we return a stream (in this case, it's using DurableAgent, but any stream works. On the client, it uses WorkflowChatTransport to resume the stream across refreshes and disconnects. The pattern is pretty simple. You can even point claude to look at the flight booking example source code, or even point it to source code of @workflow/ai package here to see how the resumption works client side

Thank you, will dig into it and see

[vahagn-madatyan]

@vahagn-madatyan ah sorry I didn't mean to say "closing your stream". I meant to say you should release the lock. that's my bad. in a step, you can release the lock on the stream after you're done writing to it. that should be enough to let the function end properly without timing out, but the stream won't be closed and other steps in the workflow can continue writing to it. The stream will automatically get closed at the end of the run (or you can always have a final step that closes it manually too)

Got you, yes I will need to check the code again, I thought i am releasing it properly. Will compare