[SPARK-48116][INFRA] Run pyspark-pandas* only in PR builder and Daily Python CIs
#46367
+137
−3
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dongjoon-hyun
changed the title
pyspark-pandas* tests to Daily Python CIspyspark-pandas* in PR builder and Daily Python CIs
dongjoon-hyun
changed the title
pyspark-pandas* in PR builder and Daily Python CIspyspark-pandas* only in PR builder and Daily Python CIs
|
Could you review this PR, @HyukjinKwon ? |
|
Could you review this PR, @zhengruifeng ? |
|
ASF Infra team requested GitHub Action usage control to Apache Spark PMC officially. Here is the umbrella JIRA issue to meet ASF Infra policy. |
|
Could you review this PR, @gengliangwang ? |
HyukjinKwon
reviewed
May 4, 2024
| @@ -347,12 +353,139 @@ jobs: | |||
| pyspark-core, pyspark-errors, pyspark-streaming | |||
| - >- | |||
| pyspark-mllib, pyspark-ml, pyspark-ml-connect | |||
| - >- | |||
There was a problem hiding this comment.
I'm fine with sikpping itself but wonder if we can deduplicate this by directly adding an if to the module based on the JSON input above. I can follow up.
|
LGTM,one comment but I can followup. |
|
Thank you for the suggestion. Yes, I hope to do the deduplication later~ |
|
Merged to master~ |
HyukjinKwon
added a commit
that referenced
this pull request
May 4, 2024
…condition ### What changes were proposed in this pull request? This PR is a followup of #46367 that simplifies the build and deduplicate them. ### Why are the changes needed? To fix the condition, and make it deduplicated. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Manually. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46380 from HyukjinKwon/SPARK-48116-followup. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
HyukjinKwon
added a commit
that referenced
this pull request
May 4, 2024
…ogic ### What changes were proposed in this pull request? This PR is another try of #46380 that is a followup of #46367 that simplifies the build and deduplicate them. ### Why are the changes needed? To fix the condition, and make it deduplicated. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Will test in my own fork: https://github.com/HyukjinKwon/spark/actions/runs/8948215777 ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46381 from HyukjinKwon/SPARK-48116-followup2. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
sinaiamonkar-sai
pushed a commit
to sinaiamonkar-sai/spark
that referenced
this pull request
May 5, 2024
…condition ### What changes were proposed in this pull request? This PR is a followup of apache#46367 that simplifies the build and deduplicate them. ### Why are the changes needed? To fix the condition, and make it deduplicated. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Manually. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46380 from HyukjinKwon/SPARK-48116-followup. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
sinaiamonkar-sai
pushed a commit
to sinaiamonkar-sai/spark
that referenced
this pull request
May 5, 2024
…ogic ### What changes were proposed in this pull request? This PR is another try of apache#46380 that is a followup of apache#46367 that simplifies the build and deduplicate them. ### Why are the changes needed? To fix the condition, and make it deduplicated. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Will test in my own fork: https://github.com/HyukjinKwon/spark/actions/runs/8948215777 ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46381 from HyukjinKwon/SPARK-48116-followup2. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
|
late LGTM, thanks @dongjoon-hyun |
|
Thanks, @zhengruifeng |
JacobZheng0927
pushed a commit
to JacobZheng0927/spark
that referenced
this pull request
May 11, 2024
…ly Python CIs ### What changes were proposed in this pull request? This PR aims to run `pyspark-pandas*` only in PR builder and Daily Python CIs. In other words, only the commit builder will skip it by default. Please note that all PR builders is not consuming ASF resources and they provides lots of test coverage everyday. - https://github.com/apache/spark/actions/workflows/build_python.yml ### Why are the changes needed? To reduce GitHub Action usage to meet ASF INFRA policy. - https://infra.apache.org/github-actions-policy.html > All workflows MUST have a job concurrency level less than or equal to 20. This means a workflow cannot have more than 20 jobs running at the same time across all matrices. Although `pandas` is an **optional** package in PySpark, this is essential for PySpark users and we have **6 test pipelines** which requires lots of resources. We need to optimize the job concurrently level to `less than or equal to 20` while keeping the test capability as much as possible. https://github.com/apache/spark/blob/f450272a9aac812d735eb5f741eec1f6cf1c837c/dev/requirements.txt#L4-L8 - pyspark-pandas - pyspark-pandas-slow - pyspark-pandas-connect-part0 - pyspark-pandas-connect-part1 - pyspark-pandas-connect-part2 - pyspark-pandas-connect-part3 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46367 from dongjoon-hyun/SPARK-48116. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
JacobZheng0927
pushed a commit
to JacobZheng0927/spark
that referenced
this pull request
May 11, 2024
…condition ### What changes were proposed in this pull request? This PR is a followup of apache#46367 that simplifies the build and deduplicate them. ### Why are the changes needed? To fix the condition, and make it deduplicated. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Manually. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46380 from HyukjinKwon/SPARK-48116-followup. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
JacobZheng0927
pushed a commit
to JacobZheng0927/spark
that referenced
this pull request
May 11, 2024
…ogic ### What changes were proposed in this pull request? This PR is another try of apache#46380 that is a followup of apache#46367 that simplifies the build and deduplicate them. ### Why are the changes needed? To fix the condition, and make it deduplicated. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? Will test in my own fork: https://github.com/HyukjinKwon/spark/actions/runs/8948215777 ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46381 from HyukjinKwon/SPARK-48116-followup2. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR aims to run
pyspark-pandas*only in PR builder and Daily Python CIs. In other words, only the commit builder will skip it by default. Please note that all PR builders is not consuming ASF resources and they provides lots of test coverage everyday.Why are the changes needed?
To reduce GitHub Action usage to meet ASF INFRA policy.
https://infra.apache.org/github-actions-policy.html
Although
pandasis an optional package in PySpark, this is essential for PySpark users and we have 6 test pipelines which requires lots of resources. We need to optimize the job concurrently level toless than or equal to 20while keeping the test capability as much as possible.spark/dev/requirements.txt
Lines 4 to 8 in f450272
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Manual review.
Was this patch authored or co-authored using generative AI tooling?
No.