Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-42452][BUILD] Remove hadoop-2 profile from Apache Spark 3.5.0 #40788

Closed
wants to merge 3 commits into from
Closed

[SPARK-42452][BUILD] Remove hadoop-2 profile from Apache Spark 3.5.0 #40788

wants to merge 3 commits into from

Conversation

LuciferYang
Copy link
Contributor

@LuciferYang LuciferYang commented Apr 14, 2023 •
edited

What changes were proposed in this pull request?

This pr aims to remove hadoop-2 profile from Apache Spark 3.5.0.

Why are the changes needed?

Spark 3.4.0 no longer releases Hadoop2 binary distribtuion(SPARK-42447) and Hadoop 2 GitHub Action job already removed after SPARK-42447, we can remove hadoop-2 profile from Apache Spark 3.5.0.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions

@LuciferYang LuciferYang marked this pull request as draft April 14, 2023 03:21
@LuciferYang LuciferYang changed the title [SPARK-42452][BUILD] Cleanup hadoop-2 profile [SPARK-42452][BUILD] Remove hadoop-2 profile from Apache Spark 3.5.0 Apr 14, 2023
@LuciferYang
Copy link
Contributor Author

Test first, need wait until 3.4 release at least

dongjoon-hyun
dongjoon-hyun reviewed Apr 14, 2023
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @LuciferYang .

@dongjoon-hyun
Copy link
Member

Also, cc FYI, @sunchao

@bjornjorgensen
Copy link
Contributor

bjornjorgensen commented Apr 14, 2023 •
edited

Test first, need wait until 3.4 release at least

3.4.0 is uploaded to mirrors.

@LuciferYang
Copy link
Contributor Author

Test first, need wait until 3.4 release at least

3.4.0 is uploaded to mirrors.

I know, but let's wait for the official website update

@LuciferYang LuciferYang marked this pull request as ready for review April 14, 2023 16:04
@sunchao
Copy link
Member

sunchao commented Apr 14, 2023

Personally I like this move, but not sure whether there are other users in the community that still depend on Hadoop 2.x. Has this been discussed before?

@bjornjorgensen
Copy link
Contributor

Personally I like this move, but not sure whether there are other users in the community that still depend on Hadoop 2.x. Has this been discussed before?

yes, like this one Dropping Apache Spark Hadoop2 Binary Distribution?

@sunchao
Copy link
Member

sunchao commented Apr 14, 2023

Oh cool, it's good then. Forgot about this thread even though I replied on it ..

dongjoon-hyun
dongjoon-hyun approved these changes Apr 14, 2023
View reviewed changes
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM (from my side)

@sunchao
Copy link
Member

sunchao commented Apr 15, 2023

Just sent an email to the community in case anyone still have concerns on this.

@LuciferYang
Copy link
Contributor Author

friendly ping @dongjoon-hyun @sunchao

@sunchao sunchao closed this in 816ebac Apr 18, 2023
@sunchao
Copy link
Member

sunchao commented Apr 18, 2023

Merged to master, thanks!

@LuciferYang
Copy link
Contributor Author

Thanks @sunchao @dongjoon-hyun @bjornjorgensen ~

@dongjoon-hyun dongjoon-hyun mentioned this pull request Mar 25, 2024
Closed
dongjoon-hyun added a commit that referenced this pull request Mar 25, 2024
@dongjoon-hyun
[SPARK-47548][BUILD] Remove unused commons-beanutils dependency
90d5065 
### What changes were proposed in this pull request?

This PR aims to remove unused `commons-beanutils` dependency from `pom.xml` and `LICENSE-binary`.

### Why are the changes needed?

#30701 removed `commons-beanutils` from `hadoop-3` profile at Apache Spark 3.2.0.
- #30701

#40788 removed `hadoop-2` profile from Apache Spark 3.5.0
- #40788

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #45705 from dongjoon-hyun/SPARK-47548.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
sweisdb pushed a commit to sweisdb/spark that referenced this pull request Apr 1, 2024
@dongjoon-hyun @sweisdb
[SPARK-47548][BUILD] Remove unused commons-beanutils dependency
d1bcc40 
### What changes were proposed in this pull request?

This PR aims to remove unused `commons-beanutils` dependency from `pom.xml` and `LICENSE-binary`.

### Why are the changes needed?

apache#30701 removed `commons-beanutils` from `hadoop-3` profile at Apache Spark 3.2.0.
- apache#30701

apache#40788 removed `hadoop-2` profile from Apache Spark 3.5.0
- apache#40788

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#45705 from dongjoon-hyun/SPARK-47548.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sunchao sunchao sunchao approved these changes

@dongjoon-hyun dongjoon-hyun dongjoon-hyun approved these changes

@bjornjorgensen bjornjorgensen bjornjorgensen approved these changes

Assignees
No one assigned
Labels
BUILD CORE DOCS INFRA KUBERNETES PYTHON SQL YARN
Projects
None yet
Milestone
No milestone
4 participants
@LuciferYang @dongjoon-hyun @bjornjorgensen @sunchao