-
-
Notifications
You must be signed in to change notification settings - Fork 745
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Workflows have large performance (latency) overhead #3712
Comments
Did a little more debugging by enabling debug in all sections in Parent workflow
Sub workflow 1
LogsSo i started checking logs between
If you notice there is a large gap (5s) from Investigation in codeIt looks like this 5s pause is coming from the Query class here: It appears to be a second sleep that's why |
This will probably be solved by the notes here: https://docs.stackstorm.com/troubleshooting/mistral.html
|
@humblearner So, using the config shown in that patch (even without the updated code to tweak the queue sleep times) i was able to reduce our workflow executions by a large amount: old
new
I'm closing this an calling it a success! Thanks @m4dcoder ! |
@nmaludy Since you're affected, just FYI #3776 we're going to fine-tune the You still can adjust the settings in your See #3776 and #3756 (comment) and would like to know if you have any comments there. |
Problem
I've been noticing that calling mistral workflows there is a fairly significant latency on the order of several seconds per workflow.
Scenario
In the reproduction we're going to have a "parent" workflow with 3x tasks each calling a "sub" workflow. The sub workflow is going to have a single task that runs a
core.local
. The command we're going to runcore.local
is thesleep
command.Reproduction
examples/actions/nick_sub_workflow.yaml
examples/actions/workflows/nick_sub_workflow.yaml
examples/actions/nick_parent_workflow.yaml
examples/actions/workflows/nick_parent_workflow.yaml
Results - Parent
Results - Sub Workflow 1
Results - Sub Workflow 2
Results - Sub Workflow 3
Observations
As you can see, each "sub" workflow takes 5s to run, even though the single task only takes 1s to run.
These runs are using the following settings in
/etc/st2/st2.conf
The text was updated successfully, but these errors were encountered: