Hi,
I have observed a regularly increasing drift in a daily scheduled workflow on GitHub runners. I know that the scheduled time is a minimal time and the workflow starts some time after the scheduled time. However, in the past few months, there is an increasing drift with an average delay of several hours, more than 4 hours now in some cases, depending on the time of the day.
Is it normal, expected, universal? If yes, what is the reason and is there anything planned to improve the situation? If not normal, what could be wrong with that job?
The job has been in place for years. This is the "nightly build" job of an open source project (ref at end of message). It builds and tests the project on Ubuntu and Windows and publishes the binaries in the project's web site. The duration of the job is approximately 1 hour and 40 minutes, except when the repository hasn't been updated in the last 24 hours, in which case it runs just for a few seconds (nothing new to build).
The job was initially scheduled at 00:40 UTC to avoid peak traffic at 00:00. Several years ago, the job used to start within 10-20 minutes. This is just from my recall because the logs are retained for the last 410 days only. From the current logs, in 2025, the average delay was 1h 40mn, already quite a lot, but stable. Since early 2026, the delay is regularly increasing, reaching 4h 30mn now.
This topic was already discussed in https://github.com/orgs/community/discussions/196910 but it seems that it now deserves opening an issue.
Several people expressed similar feelings of increasing delays, although not of the same magnitude of more than 4 hours.
The appropriate time of the day was discussed, suggesting to get away from the "midnight UTC bottleneck", a favorite time for many developers, without relation to their actual time zone. Various times of the day were tested with mixed results, but still more than one or two hours delay:
- 00:40 UTC (initial schedule): from 1h30 to 4h30 delayed over a year.
- 14:37 UTC: 3h delayed.
- 23:11 UTC: 1h to 2h delayed.
Using a Python script, I collect the created and start time of each run over the last 410 saved schedule. See a partial output below. Note that created and start times are always identical and I didn't find any other relevant time in the GitHub API.
Modified schedule at 23:11 UTC:
Run# Created Started Duration Status Origin
------ ------------------- ------------------- -------- -------- ---------
2376 2026-06-02 01:35:42 2026-06-02 01:35:42 00:00:14 success schedule
2375 2026-06-01 00:14:44 2026-06-01 00:14:44 00:00:11 success schedule
2374 2026-05-31 00:12:59 2026-05-31 00:12:59 01:25:49 success schedule
Modified schedule at 14:37 UTC:
2373 2026-05-29 17:53:36 2026-05-29 17:53:36 01:23:39 success schedule
2372 2026-05-28 17:50:14 2026-05-28 17:50:14 01:52:34 success schedule
2371 2026-05-27 17:42:39 2026-05-27 20:13:00 00:02:06 failure schedule
2370 2026-05-26 17:46:12 2026-05-26 17:46:12 01:47:34 success schedule
Initial schedule at 00:40 UTC:
2369 2026-05-25 05:08:22 2026-05-25 05:08:22 00:59:21 failure schedule
2368 2026-05-24 04:49:20 2026-05-24 04:49:20 01:50:03 success schedule
...
2315 2026-04-01 03:52:31 2026-04-01 03:52:31 00:00:12 success schedule
...
2283 2026-03-01 03:32:43 2026-03-01 03:32:43 00:00:12 success schedule
...
2255 2026-02-01 03:39:12 2026-02-01 03:39:12 00:00:13 success schedule
...
2224 2026-01-01 02:57:05 2026-01-01 02:57:05 00:00:14 success schedule
...
2193 2025-12-01 02:57:02 2025-12-01 02:57:02 01:47:27 success schedule
...
2160 2025-11-01 02:22:19 2025-11-01 02:22:19 00:00:10 success schedule
...
2124 2025-10-01 02:21:29 2025-10-01 02:21:29 00:00:14 success schedule
...
2094 2025-09-01 02:31:11 2025-09-01 02:31:11 00:00:11 success schedule
...
2061 2025-08-01 03:00:41 2025-08-01 03:00:41 00:00:16 success schedule
...
2029 2025-07-01 02:39:34 2025-07-01 02:39:34 00:00:11 success schedule
...
1993 2025-06-01 02:50:41 2025-06-01 02:50:41 01:32:13 success schedule
...
1962 2025-05-01 02:27:41 2025-05-01 02:27:41 01:30:04 success schedule
...
1947 2025-04-18 02:12:35 2025-04-18 02:12:35 00:00:35 success schedule
Project: https://github.com/tsduck/tsduck
Workflow: https://github.com/tsduck/tsduck/blob/master/.github/workflows/nightly-build.yml
Access to workflow runs: https://github.com/tsduck/tsduck/actions/workflows/nightly-build.yml
Python script to get the run times: https://github.com/tsduck/tsduck/blob/master/pkg/github/nightly-build-statistics.py
Hi,
I have observed a regularly increasing drift in a daily scheduled workflow on GitHub runners. I know that the scheduled time is a minimal time and the workflow starts some time after the scheduled time. However, in the past few months, there is an increasing drift with an average delay of several hours, more than 4 hours now in some cases, depending on the time of the day.
Is it normal, expected, universal? If yes, what is the reason and is there anything planned to improve the situation? If not normal, what could be wrong with that job?
The job has been in place for years. This is the "nightly build" job of an open source project (ref at end of message). It builds and tests the project on Ubuntu and Windows and publishes the binaries in the project's web site. The duration of the job is approximately 1 hour and 40 minutes, except when the repository hasn't been updated in the last 24 hours, in which case it runs just for a few seconds (nothing new to build).
The job was initially scheduled at 00:40 UTC to avoid peak traffic at 00:00. Several years ago, the job used to start within 10-20 minutes. This is just from my recall because the logs are retained for the last 410 days only. From the current logs, in 2025, the average delay was 1h 40mn, already quite a lot, but stable. Since early 2026, the delay is regularly increasing, reaching 4h 30mn now.
This topic was already discussed in https://github.com/orgs/community/discussions/196910 but it seems that it now deserves opening an issue.
Several people expressed similar feelings of increasing delays, although not of the same magnitude of more than 4 hours.
The appropriate time of the day was discussed, suggesting to get away from the "midnight UTC bottleneck", a favorite time for many developers, without relation to their actual time zone. Various times of the day were tested with mixed results, but still more than one or two hours delay:
Using a Python script, I collect the created and start time of each run over the last 410 saved schedule. See a partial output below. Note that created and start times are always identical and I didn't find any other relevant time in the GitHub API.
Modified schedule at 23:11 UTC:
Modified schedule at 14:37 UTC:
Initial schedule at 00:40 UTC:
Project: https://github.com/tsduck/tsduck
Workflow: https://github.com/tsduck/tsduck/blob/master/.github/workflows/nightly-build.yml
Access to workflow runs: https://github.com/tsduck/tsduck/actions/workflows/nightly-build.yml
Python script to get the run times: https://github.com/tsduck/tsduck/blob/master/pkg/github/nightly-build-statistics.py