Move auto-refresh interval to config variable

[rachelwigell]

Description

Auto-refresh is an awesome new feature! Right now it occurs every 3 seconds, and this interval is not configurable. Making this interval configurable would be useful.

Use case/motivation

On a DAG with many tasks and many active DAG runs, auto-refresh can be a significant strain on the webserver. We observed our production webserver reach 100% CPU today when we had about 30 running DAG runs of a DAG containing about 50 tasks. Being able to increase the interval to guard against this would be helpful.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[ashb]

The config option is fine, but the number of Tasks in a DAG shouldn't have any bearing on the number of requests made (it should be a single request per use on a Graph or Tree View page).

[rachelwigell]

Hmm okay. We were able to observe pretty clearly that when we had tree view open, with auto-refresh on, looking back over 365 DAG runs, of which about 30 were in the running state, our webserver ground to a halt and CPU was at 100%. When we closed that tab it recovered. I can do some experimentation in my dev environment tonight to see if I can nail down what exactly strains the CPU so much.

[rachelwigell]

Yeah I haven't quite been able to locally replicate the same severe effects on CPU we saw a few days ago. I do see that we are indeed making just one request to populate the tree view regardless of whether I have many active tasks or DAG runs. Of course, that request has to do more work if there are lots of tasks to iterate over, but the n's here don't seem like enough to tank a server. Regardless, I'll move forward with my PR as I think folks still might like being able to customize this setting.

[ashb]

Yeah, absolutely no problem with your PR, I was just a bit surprised by the effects!