Scheduled Workflows delayed by up to 6h (even for self-hosted runners) #213505
Replies: 4 comments
|
💬 Your Product Feedback Has Been Submitted 🎉 Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users. Here's what you can expect moving forward ⏩
Where to look to see what's shipping 👀
What you can do in the meantime 💻
As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities. Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐ |
|
One detail in your description rules out the runners: the delay happens before a run record exists. If a job is sitting in GitHub documents scheduled workflows as best-effort and explicitly says they can be delayed during high load. Daily crons funnel into a narrow window (midnight UTC is notorious), so Monday spikes are consistent with scheduler contention rather than anything in your repo. The 60-day-inactivity auto-disable doesn't apply here; xsuite is clearly active. What I'd do with that plot:
I've seen teams accept the delay for nightly builds and move only their time-sensitive jobs off GitHub. If this is a nightly check with no hard deadline, that may be the sane trade-off; if it feeds something time-critical, external dispatch is the only reliable path today. |
|
The public run and jobs APIs let you split the delay into three intervals: expected cron time to run For example, public run Use |
|
A workaround you could try while this is open: stop depending on
Also, if your crons sit exactly on None of this fixes the underlying problem on GitHub's side, but it gives you predictable timing in the meantime. |
Uh oh!
There was an error while loading. Please reload this page.
🏷️ Discussion Type
Bug
💬 Feature/Topic Area
ARC (Actions Runner Controller)
Discussion Details
This repeats what many other open (unanswered) issues already point out, that since late August there has been a huge degradation in scheduled workflows in that they start running up to 6h or more after the original scheduled time. This seemingly also happens even if the workflow is supposed to run on a self-hosted runner (which in principle should not put as much strain on GH's infrastructure).
We have been observing this in particular in xsuite/xsuite, with the below plot showing delays for various daily workflows over the past couple of months:
As can be see from the plot, the situation is clearly deteriorating, and in particular you can see the spikes on each Monday, which might mean that the scheduler is not very robust?
All reactions