[SPARK-48149][INFRA] Serialize build_python.yml to run a single Python version per cron schedule
#46407
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…hon version per cron schedule
|
Could you review this PR, @HyukjinKwon ? |
dongjoon-hyun
commented
May 6, 2024
| include: | ||
| - pyversion: ${{ github.event.schedule == '0 15 * * *' && "pypy3" }} | ||
| - pyversion: ${{ github.event.schedule == '0 17 * * *' && "python3.10" }} | ||
| - pyversion: ${{ github.event.schedule == '0 19 * * *' && "python3.12" }} |
There was a problem hiding this comment.
dongjoon-hyun
commented
May 6, 2024
| name: "Build / Python-only (master, PyPy 3.9/Python 3.10/Python 3.12)" | ||
|
|
||
| on: | ||
| schedule: | ||
| - cron: '0 15 * * *' | ||
| - cron: '0 17 * * *' | ||
| - cron: '0 19 * * *' |
There was a problem hiding this comment.
Although we can have multiple files like build_python3.10.yml, build_python3.12.yml, .... , I guess you prefer to keep this in a single file in order to avoid Duplication, @HyukjinKwon .
Please let me know if we can duplicate this file simply per Python version.
HyukjinKwon
approved these changes
May 6, 2024
|
Thank you, @HyukjinKwon ! Then, let's try this one first. :) |
|
Merged to master. |
HyukjinKwon
pushed a commit
that referenced
this pull request
May 8, 2024
### What changes were proposed in this pull request? This is a follow-up of - #46407 ### Why are the changes needed? To fix the invalid syntax error - https://github.com/apache/spark/actions/runs/8989374763 > The workflow is not valid. .github/workflows/build_python.yml (Line: 37, Col: 24): Unexpected symbol: '"pypy3"'. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46454 from dongjoon-hyun/SPARK-48149-2. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
dongjoon-hyun
added a commit
that referenced
this pull request
May 8, 2024
### What changes were proposed in this pull request? This PR aims to split `build_python.yml` into per-version cron jobs. Technically, this includes a revert of SPARK-48149 and choose [the discussed alternative](#46407 (comment)). - #46407 - #46454 ### Why are the changes needed? To recover Python CI successfully in ASF INFRA policy. - https://github.com/apache/spark/actions/workflows/build_python.yml ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46477 from dongjoon-hyun/SPARK-48200. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
JacobZheng0927
pushed a commit
to JacobZheng0927/spark
that referenced
this pull request
May 11, 2024
…hon version per cron schedule ### What changes were proposed in this pull request? This PR aims to serialize `build_python.yml` to run a single Python version per cron schedule to reduce the maximum concurrency per single GitHub Action job. ### Why are the changes needed? Currently, `build_python.yml` triggers 60 jobs. `30` of `60` jobs are running concurrently because 10 test pipelines are required per Python version. - https://github.com/apache/spark/actions/workflows/build_python.yml <img width="731" alt="Screenshot 2024-05-06 at 14 28 08" src="https://github.com/apache/spark/assets/9700541/e4f4e9d2-2b2e-43b9-a760-6b9943c7b5b7"> According to https://infra.apache.org/github-actions-policy.html, > All workflows SHOULD have a job concurrency level less than or equal to 15. After this PR, the maximum concurrently level will be 10. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review because this is a daily CI. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46407 from dongjoon-hyun/SPARK-48149. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
JacobZheng0927
pushed a commit
to JacobZheng0927/spark
that referenced
this pull request
May 11, 2024
### What changes were proposed in this pull request? This is a follow-up of - apache#46407 ### Why are the changes needed? To fix the invalid syntax error - https://github.com/apache/spark/actions/runs/8989374763 > The workflow is not valid. .github/workflows/build_python.yml (Line: 37, Col: 24): Unexpected symbol: '"pypy3"'. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46454 from dongjoon-hyun/SPARK-48149-2. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
JacobZheng0927
pushed a commit
to JacobZheng0927/spark
that referenced
this pull request
May 11, 2024
### What changes were proposed in this pull request? This PR aims to split `build_python.yml` into per-version cron jobs. Technically, this includes a revert of SPARK-48149 and choose [the discussed alternative](apache#46407 (comment)). - apache#46407 - apache#46454 ### Why are the changes needed? To recover Python CI successfully in ASF INFRA policy. - https://github.com/apache/spark/actions/workflows/build_python.yml ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46477 from dongjoon-hyun/SPARK-48200. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR aims to serialize
build_python.ymlto run a single Python version per cron schedule to reduce the maximum concurrency per single GitHub Action job.Why are the changes needed?
Currently,
build_python.ymltriggers 60 jobs.30of60jobs are running concurrently because 10 test pipelines are required per Python version.According to https://infra.apache.org/github-actions-policy.html,
After this PR, the maximum concurrently level will be 10.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Manual review because this is a daily CI.
Was this patch authored or co-authored using generative AI tooling?
No.