Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some Durable Functions instances stay in Pending state when tested at scale, and scale stops at 7 servers #412

Closed
nzthiago opened this issue Aug 27, 2018 · 3 comments
Labels
Needs: Investigation 🔍 A deeper investigation needs to be done by the project maintainers.

Comments

@nzthiago
Copy link
Member

While concurrently running hundreds of identical orchestrations (600 instances in our case) using Azure Durable Functions, a customer has observed that the server failed to scale beyond seven instances and some insteances stay in the Pending state.

Other runs yielded similar results. Here are the results of one of the runs:

  • Around 400 of them completed with Success (top level Try Catch didn’t catch an exception).
  • Around 10-20 of them completed with Fail ( top level Try Catch caught an exception)
  • 2 of them Failed with RuntimeStatus = Fail
  • 2 of them are still Pending
  • Around 150 were still Running

Pending orchestrations for Durable Functions should eventually run rather than remain stuck in a pending state, and servers should scale beyond 7.

@cgillum cgillum added the Needs: Investigation 🔍 A deeper investigation needs to be done by the project maintainers. label Aug 27, 2018
@nzthiago
Copy link
Member Author

Update: all the tests were done using Consumption Plan

@nzthiago
Copy link
Member Author

Another note; after 3 days from a total of 600 parent instances there were 88 Running, 2 Pending and 2 Failed parent instances (didn’t count the subs instantiated). There are no more queue messages in any of the control queues, so it looks like all the running and pending ones are orphans at this point.

@cgillum
Copy link
Collaborator

cgillum commented Oct 29, 2018

Closing this as a duplicate of #460.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Needs: Investigation 🔍 A deeper investigation needs to be done by the project maintainers.
Projects
None yet
Development

No branches or pull requests

2 participants