Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add a timeout to CI builds #103180

Closed
JelleZijlstra opened this issue Apr 2, 2023 · 5 comments
Closed

Add a timeout to CI builds #103180

JelleZijlstra opened this issue Apr 2, 2023 · 5 comments
Assignees
@sobolevn
Labels
build The build process and cross-build

Comments

@JelleZijlstra
Copy link
Member

JelleZijlstra commented Apr 2, 2023 •
edited by bedevere-bot

The joke PR #103178 included some very slow code that runs at import time, and CI took >3 hours before I canceled it. This wastes resources and is a possible abuse vector; a malicious actor could use it to eat up our CI resource quota. We should have some reasonable timeout (1 hour?) for all CI builds.

Linked PRs
@JelleZijlstra JelleZijlstra added the build The build process and cross-build label Apr 2, 2023
@kylehofmann
Copy link

I would like to publicly apologize for the PR. It didn't turn out the way I had intended.

@JelleZijlstra
Copy link
Member Author

No apology necessary, it's better that we find out about this problem through a well-intentioned joke.

I was actually surprised when I saw the PR that it showed you as a first-time contributor here. Based on the level of your contributions on the Python Discord, I'm sure there's lots of useful work you could do on CPython if you are so inclined. Perhaps your next PR will be less quadratic. (And if you're not interested or don't have time, that's perfectly fine too, of course.)

@arhadthedev
Copy link
Member

Related issue (for the build step only): gh-90361.

@sobolevn
Copy link
Member

I had some time to look at this. I think that we need to base timeouts:

  1. For short actions, like stale and labels
  2. For build / test actions

Short timeout

stale takes ~1m average to finish:
Снимок экрана 2023-04-11 в 12 10 11

new-bugs-notifier takes several seconds:
Снимок экрана 2023-04-11 в 12 11 46

check-labels also takes several seconds:
Снимок экрана 2023-04-11 в 12 12 27

verify-ensurepip-wheels takes several seconds:
Снимок экрана 2023-04-11 в 12 13 34

project-updater takes several seconds as well:
Снимок экрана 2023-04-11 в 12 15 02

I think that 10 minutes as a timeout is a reasonable number for this group.
It will allow quite some performance degradation, but still keep our resources under controll.

Slow actions

docs can work from 10m up to 20m:
Снимок экрана 2023-04-11 в 12 17 05

testsmsi can take from several minutes up to 20m:
Снимок экрана 2023-04-11 в 12 18 26

build takes around 30m:
Снимок экрана 2023-04-11 в 12 19 24

So, using 1h is a resonable timeout.

I will send a PR with these timeouts in a moment.
Plus, I will change per-step timeouts from #90361 to be job-level (and increase them).

sobolevn added a commit to sobolevn/cpython that referenced this issue Apr 11, 2023
@sobolevn
pythongh-103180: Add CI timeouts to all GtiHub Actions jobs
66f638f
@sobolevn sobolevn mentioned this issue Apr 11, 2023
Merged
@sobolevn sobolevn self-assigned this Apr 11, 2023
hugovk pushed a commit that referenced this issue Apr 14, 2023
@sobolevn
gh-103180: Add CI timeouts to all GitHub Actions jobs (#103437)
be8903e
carljm added a commit to carljm/cpython that referenced this issue Apr 14, 2023
@carljm
Merge branch 'main' into superopt
64da49f 
* main:
  pythongh-103532: Remove TKINTER_PROTECT_LOADTK code (pythonGH-103535)
  pythongh-103180: Add CI timeouts to all GitHub Actions jobs (python#103437)
  Remove double space in import error message (python#103458)
  ipaddress: Remove non-existent ip_str param from docstring (python#103461)
  Fix syntax typo in isolating extensions doc (python#103516)
  pythongh-103406: Modernize pos-only arguments usage in `test_signature` (python#103407)
  Proofread howto/perf_profiling.rst (python#103530)
  Fix unused functions warnings in instrumentation.c (pythonGH-103515)
sobolevn added a commit to sobolevn/cpython that referenced this issue Apr 14, 2023
@sobolevn
[3.11] pythongh-103180: Add CI timeouts to all GitHub Actions jobs (p…
f7e8ca1 
…ythonGH-103437).

(cherry picked from commit be8903e)

Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
@bedevere-bot bedevere-bot mentioned this issue Apr 14, 2023
Merged
carljm added a commit to carljm/cpython that referenced this issue Apr 14, 2023
@carljm
Merge branch 'superopt' into superopt_spec
b73cff0 
* superopt:
  fix incompatible types
  update generated cases
  don't unnecessarily re-find args in error case
  Apply suggestions from code review
  pythongh-103532: Remove TKINTER_PROTECT_LOADTK code (pythonGH-103535)
  pythongh-103180: Add CI timeouts to all GitHub Actions jobs (python#103437)
  Remove double space in import error message (python#103458)
  ipaddress: Remove non-existent ip_str param from docstring (python#103461)
  Fix syntax typo in isolating extensions doc (python#103516)
  pythongh-103406: Modernize pos-only arguments usage in `test_signature` (python#103407)
  Proofread howto/perf_profiling.rst (python#103530)
  Fix unused functions warnings in instrumentation.c (pythonGH-103515)
hugovk pushed a commit that referenced this issue Apr 15, 2023
@sobolevn
[3.11] gh-103180: Add CI timeouts to all GitHub Actions jobs (GH-103437…
a3242c7 
…). (#103543)
@hauntsaninja
Copy link
Contributor

Looks like this was completed, thanks all!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sobolevn sobolevn

Labels
build The build process and cross-build
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@JelleZijlstra @sobolevn @arhadthedev @kylehofmann @hauntsaninja