-
Notifications
You must be signed in to change notification settings - Fork 28.9k
[SPARK-54426][INFRA] Fix release-build.sh to detect REPO_ID correctly
#53136
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Could you review this PR, @HyukjinKwon ? |
|
Thanks! |
|
Thank you, @HyukjinKwon . Merged to master/4.1. |
…ctly ### What changes were proposed in this pull request? This PR aims to fix `release-build.sh` to detect `REPO_ID` correctly. ### Why are the changes needed? Previously, we use `grep -A 5` to find `description` tag. https://github.com/apache/spark/blob/f328b5ef14c9ef4e2d04ab69c0578ab461388715/dev/create-release/release-build.sh#L501-L502 However, it's insufficient as of now. According to Today's result, we need to grep 13 lines like the following. ``` $ curl --retry 10 --retry-all-errors -s -u "$ASF_USERNAME:$ASF_PASSWORD" https://repository.apache.org/service/local/staging/profile_repositories | grep -A 5 "<repositoryId>orgapachespark-" <repositoryId>orgapachespark-1505</repositoryId> <type>closed</type> <policy>release</policy> <userId>dongjoon</userId> <userAgent>curl/7.81.0</userAgent> <ipAddress>35.94.112.49</ipAddress> ``` ``` $ curl --retry 10 --retry-all-errors -s -u "$ASF_USERNAME:$ASF_PASSWORD" https://repository.apache.org/service/local/staging/profile_repositories | grep -A 13 "<repositoryId>orgapachespark-" <repositoryId>orgapachespark-1505</repositoryId> <type>closed</type> <policy>release</policy> <userId>dongjoon</userId> <userAgent>curl/7.81.0</userAgent> <ipAddress>35.94.112.49</ipAddress> <repositoryURI>https://repository.apache.org/content/repositories/orgapachespark-1505</repositoryURI> <created>2025-11-16T20:23:35.413Z</created> <createdDate>Sun Nov 16 20:23:35 UTC 2025</createdDate> <createdTimestamp>1763324615413</createdTimestamp> <updated>2025-11-16T21:02:45.041Z</updated> <updatedDate>Sun Nov 16 21:02:45 UTC 2025</updatedDate> <updatedTimestamp>1763326965041</updatedTimestamp> <description>Apache Spark 4.1.0-preview4 (commit c125aea)</description> ``` ### Does this PR introduce _any_ user-facing change? No behavior change. ### How was this patch tested? Manually test. ``` $ curl --retry 10 --retry-all-errors -s -u "$ASF_USERNAME:$ASF_PASSWORD" https://repository.apache.org/service/local/staging/profile_repositories | grep -A 13 "<repositoryId>orgapachespark-" | awk '/<repositoryId>/ { id = $0 } /<description>/ && $0 ~ /Apache Spark '"$RELEASE_VERSION"'/ { print id }' <repositoryId>orgapachespark-1505</repositoryId> ``` After merging this, I'm going to test with `Apache Spark 4.1.0-preview4`. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #53136 from dongjoon-hyun/SPARK-54426. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit be281fb) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
|
Let me cherry-pick this to branch-4 and branch-3.5 too. It should work in the same way |
…ctly ### What changes were proposed in this pull request? This PR aims to fix `release-build.sh` to detect `REPO_ID` correctly. ### Why are the changes needed? Previously, we use `grep -A 5` to find `description` tag. https://github.com/apache/spark/blob/f328b5ef14c9ef4e2d04ab69c0578ab461388715/dev/create-release/release-build.sh#L501-L502 However, it's insufficient as of now. According to Today's result, we need to grep 13 lines like the following. ``` $ curl --retry 10 --retry-all-errors -s -u "$ASF_USERNAME:$ASF_PASSWORD" https://repository.apache.org/service/local/staging/profile_repositories | grep -A 5 "<repositoryId>orgapachespark-" <repositoryId>orgapachespark-1505</repositoryId> <type>closed</type> <policy>release</policy> <userId>dongjoon</userId> <userAgent>curl/7.81.0</userAgent> <ipAddress>35.94.112.49</ipAddress> ``` ``` $ curl --retry 10 --retry-all-errors -s -u "$ASF_USERNAME:$ASF_PASSWORD" https://repository.apache.org/service/local/staging/profile_repositories | grep -A 13 "<repositoryId>orgapachespark-" <repositoryId>orgapachespark-1505</repositoryId> <type>closed</type> <policy>release</policy> <userId>dongjoon</userId> <userAgent>curl/7.81.0</userAgent> <ipAddress>35.94.112.49</ipAddress> <repositoryURI>https://repository.apache.org/content/repositories/orgapachespark-1505</repositoryURI> <created>2025-11-16T20:23:35.413Z</created> <createdDate>Sun Nov 16 20:23:35 UTC 2025</createdDate> <createdTimestamp>1763324615413</createdTimestamp> <updated>2025-11-16T21:02:45.041Z</updated> <updatedDate>Sun Nov 16 21:02:45 UTC 2025</updatedDate> <updatedTimestamp>1763326965041</updatedTimestamp> <description>Apache Spark 4.1.0-preview4 (commit c125aea)</description> ``` ### Does this PR introduce _any_ user-facing change? No behavior change. ### How was this patch tested? Manually test. ``` $ curl --retry 10 --retry-all-errors -s -u "$ASF_USERNAME:$ASF_PASSWORD" https://repository.apache.org/service/local/staging/profile_repositories | grep -A 13 "<repositoryId>orgapachespark-" | awk '/<repositoryId>/ { id = $0 } /<description>/ && $0 ~ /Apache Spark '"$RELEASE_VERSION"'/ { print id }' <repositoryId>orgapachespark-1505</repositoryId> ``` After merging this, I'm going to test with `Apache Spark 4.1.0-preview4`. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #53136 from dongjoon-hyun/SPARK-54426. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
…ctly This PR aims to fix `release-build.sh` to detect `REPO_ID` correctly. Previously, we use `grep -A 5` to find `description` tag. https://github.com/apache/spark/blob/f328b5ef14c9ef4e2d04ab69c0578ab461388715/dev/create-release/release-build.sh#L501-L502 However, it's insufficient as of now. According to Today's result, we need to grep 13 lines like the following. ``` $ curl --retry 10 --retry-all-errors -s -u "$ASF_USERNAME:$ASF_PASSWORD" https://repository.apache.org/service/local/staging/profile_repositories | grep -A 5 "<repositoryId>orgapachespark-" <repositoryId>orgapachespark-1505</repositoryId> <type>closed</type> <policy>release</policy> <userId>dongjoon</userId> <userAgent>curl/7.81.0</userAgent> <ipAddress>35.94.112.49</ipAddress> ``` ``` $ curl --retry 10 --retry-all-errors -s -u "$ASF_USERNAME:$ASF_PASSWORD" https://repository.apache.org/service/local/staging/profile_repositories | grep -A 13 "<repositoryId>orgapachespark-" <repositoryId>orgapachespark-1505</repositoryId> <type>closed</type> <policy>release</policy> <userId>dongjoon</userId> <userAgent>curl/7.81.0</userAgent> <ipAddress>35.94.112.49</ipAddress> <repositoryURI>https://repository.apache.org/content/repositories/orgapachespark-1505</repositoryURI> <created>2025-11-16T20:23:35.413Z</created> <createdDate>Sun Nov 16 20:23:35 UTC 2025</createdDate> <createdTimestamp>1763324615413</createdTimestamp> <updated>2025-11-16T21:02:45.041Z</updated> <updatedDate>Sun Nov 16 21:02:45 UTC 2025</updatedDate> <updatedTimestamp>1763326965041</updatedTimestamp> <description>Apache Spark 4.1.0-preview4 (commit c125aea)</description> ``` No behavior change. Manually test. ``` $ curl --retry 10 --retry-all-errors -s -u "$ASF_USERNAME:$ASF_PASSWORD" https://repository.apache.org/service/local/staging/profile_repositories | grep -A 13 "<repositoryId>orgapachespark-" | awk '/<repositoryId>/ { id = $0 } /<description>/ && $0 ~ /Apache Spark '"$RELEASE_VERSION"'/ { print id }' <repositoryId>orgapachespark-1505</repositoryId> ``` After merging this, I'm going to test with `Apache Spark 4.1.0-preview4`. No. Closes #53136 from dongjoon-hyun/SPARK-54426. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit be281fb) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
|
Oh, thanks, @HyukjinKwon ! |
|
I cancelled the release job because other issues. Initially, For the actual testing, I'll do next week with Apache Spark 4.1.0 RC and finalizing it, @HyukjinKwon . |
|
For 4.1.0-preview4, I'll proceed to finalize manually. |
|
thanks! |
What changes were proposed in this pull request?
This PR aims to fix
release-build.shto detectREPO_IDcorrectly.Why are the changes needed?
Previously, we use
grep -A 5to finddescriptiontag.spark/dev/create-release/release-build.sh
Lines 501 to 502 in f328b5e
However, it's insufficient as of now. According to Today's result, we need to grep 13 lines like the following.
Does this PR introduce any user-facing change?
No behavior change.
How was this patch tested?
Manually test.
After merging this, I'm going to test with
Apache Spark 4.1.0-preview4.Was this patch authored or co-authored using generative AI tooling?
No.