Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Check style with profiles of Spark 3.4 and 3.5 #5444

Closed
wants to merge 6 commits into from

Conversation

bowenliang123
Copy link
Contributor

@bowenliang123 bowenliang123 commented Oct 17, 2023

Why are the changes needed?

  • Check style with profile of Spark 3.4 and 3.5 in the style workflow
  • Isolated scalastyle check for Spark 3.1 profile, as Iceberg 1.3.1 does not support Spark 3.5

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before make a pull request

Was this patch authored or co-authored using generative AI tooling?

No.

@github-actions github-actions bot added the kind:infra license, community building, project builds, asf infra related, etc. label Oct 17, 2023
@bowenliang123 bowenliang123 changed the title Check style with profile of Spark 3.4 and 3.5 in CI tests Check style with profile of Spark 3.4 and 3.5 in style workflow Oct 17, 2023
@bowenliang123 bowenliang123 added this to the v1.8.0 milestone Oct 17, 2023
@bowenliang123 bowenliang123 self-assigned this Oct 17, 2023
@bowenliang123 bowenliang123 marked this pull request as draft October 17, 2023 08:38
@yaooqinn yaooqinn changed the title Check style with profile of Spark 3.4 and 3.5 in style workflow Check style with profiles of Spark 3.4 and 3.5 in style workflow Oct 17, 2023
@bowenliang123
Copy link
Contributor Author

bowenliang123 commented Oct 17, 2023

There's a blocker issue here.
Using multiple Spark profiles in style checks simultaneously, the spark-3.1 profile pollutes the Iceberg version to 1.3.1, which does not release support for Spark 3.5 and causes the Scalastyle checks to fail.

Error:  Failed to execute goal on project kyuubi-spark-sql-engine_2.12: 
Could not resolve dependencies for project org.apache.kyuubi:kyuubi-spark-sql-engine_2.12:jar:1.9.0-SNAPSHOT:
 org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:jar:1.3.1 was not found in https://maven-central-asia.storage-download.googleapis.com/maven2/ during a previous attempt.

log: https://github.com/apache/kyuubi/actions/runs/6544718529/job/17771748593?pr=5444#step:8:413

Resolve this problem by running scalastyle checks with the e Spark 3.1 profile separately.

@codecov-commenter
Copy link

codecov-commenter commented Oct 17, 2023

Codecov Report

Merging #5444 (ca98c52) into master (fdd8f17) will not change coverage.
Report is 4 commits behind head on master.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #5444   +/-   ##
======================================
  Coverage    0.00%   0.00%           
======================================
  Files         588     588           
  Lines       33450   33450           
  Branches     4400    4400           
======================================
  Misses      33450   33450           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@bowenliang123 bowenliang123 marked this pull request as ready for review October 17, 2023 13:38
@bowenliang123 bowenliang123 changed the title Check style with profiles of Spark 3.4 and 3.5 in style workflow Check style with profiles of Spark 3.4 and 3.5 Oct 17, 2023
@bowenliang123 bowenliang123 changed the title Check style with profiles of Spark 3.4 and 3.5 [CI] Check style with profiles of Spark 3.4 and 3.5 Oct 17, 2023
bowenliang123 added a commit that referenced this pull request Oct 19, 2023
### _Why are the changes needed?_

- Check style with profile of Spark 3.4 and 3.5 in the style workflow
- Isolated scalastyle check for Spark 3.1 profile, as Iceberg 1.3.1 does not support Spark 3.5

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No.

Closes #5444 from bowenliang123/spotless-spark34.

Closes #5444

ca98c52 [Bowen Liang] update
9550a4a [Bowen Liang] enable spark-3.1 profile for spotless style checks
dae3529 [Bowen Liang] revert order
3e488cf [Bowen Liang] Check with Spark 3.1 profile separately
e879f09 [Bowen Liang] revert the order of spark versions in profiles
16456a7 [Bowen Liang] check style with profile of Spark 3.4 and 3.5

Authored-by: Bowen Liang <liangbowen@gf.com.cn>
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
(cherry picked from commit 48bdc7d)
Signed-off-by: Bowen Liang <liangbowen@gf.com.cn>
@bowenliang123
Copy link
Contributor Author

Thanks, merged to master (1.9.0) and branch-1.8 (1.8.0).

@bowenliang123 bowenliang123 deleted the spotless-spark34 branch October 19, 2023 06:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:infra license, community building, project builds, asf infra related, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants