Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Python][CI] Update spark versions tested on our nightly tests #36199

Closed
raulcd opened this issue Jun 21, 2023 · 2 comments · Fixed by #36347
Closed

[Python][CI] Update spark versions tested on our nightly tests #36199

raulcd opened this issue Jun 21, 2023 · 2 comments · Fixed by #36347
Assignees
@raulcd
Labels
Component: Continuous Integration Component: Python Type: enhancement
Milestone
14.0.0

Comments

@raulcd
Copy link
Member

raulcd commented Jun 21, 2023

Describe the enhancement requested

It does seem we are using some pretty old and deprecated spark versions on our CI nightly integration builds with spark.
As discussed on this PR comment discussion: #36061 (comment)

The following comments were added:

Do we still need Spark 3.1.2?

I think that Spark 3.1 reached EOL:

https://spark.apache.org/versioning-policy.html

Feature release branches will, generally, be maintained with bug fix releases for a period of 18 months.

And 3.1.1 was released on 2021-03-02:

https://spark.apache.org/news/index.html

Spark 3.1.1 released
March 2, 2021

3.2 might reach EOL because 3.2.0 was released on 2021-10-13:

Spark 3.2.0 released
October 13, 2021

Component(s)

Continuous Integration, Python
@raulcd
Copy link
Member Author

raulcd commented Jun 21, 2023

@kiszk do you know what are the versions we should be testing?

@raulcd raulcd mentioned this issue Jun 21, 2023
Merged
@kiszk
Copy link
Member

kiszk commented Jun 22, 2023

You are right. This (at the bottom) is the maintenance policy. For 18 months from the release. Based on the policy, the current active Sparks are 3.2.x, 3.3.x, and 3.4.x.

Spark 3.2.x will be retired soon.

Spark 3.2.0 released
October 13, 2021

Spark 3.5 will be released after this summer

August 2023 | Release candidates (RC), voting, etc. until final release passes

So, my suggestion is to test Spark 3.4.0.

raulcd added a commit to raulcd/arrow that referenced this issue Jun 28, 2023
@raulcd
apacheGH-36199: [Python][CI][Spark] Update spark versions used on our…
9dff708 
… nightly tests
@github-actions github-actions bot mentioned this issue Jun 28, 2023
Merged
raulcd added a commit to raulcd/arrow that referenced this issue Aug 1, 2023
@raulcd
apacheGH-36199: [Python][CI][Spark] Update spark versions used on our…
1a71645 
… nightly tests
assignUser pushed a commit that referenced this issue Aug 1, 2023
@raulcd
GH-36199: [Python][CI][Spark] Update spark versions used on our night…
e34d4c9 
…ly tests (#36347)

### Rationale for this change

We are currently testing with very old or deprecated spark versions.

### What changes are included in this PR?

Update the spark versions to be tested.

### Are these changes tested?

They will via archery

### Are there any user-facing changes?

No
* Closes: #36199

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Jacob Wujciak-Jens <jacob@wujciak.de>
@assignUser assignUser added this to the 14.0.0 milestone Aug 1, 2023
loicalleyne pushed a commit to loicalleyne/arrow that referenced this issue Nov 13, 2023
@raulcd @loicalleyne
apacheGH-36199: [Python][CI][Spark] Update spark versions used on our…
fed2d22 
… nightly tests (apache#36347)

### Rationale for this change

We are currently testing with very old or deprecated spark versions.

### What changes are included in this PR?

Update the spark versions to be tested.

### Are these changes tested?

They will via archery

### Are there any user-facing changes?

No
* Closes: apache#36199

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Jacob Wujciak-Jens <jacob@wujciak.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@raulcd raulcd

Labels
Component: Continuous Integration Component: Python Type: enhancement
Projects
None yet
Milestone
14.0.0
Development

Successfully merging a pull request may close this issue.

GH-36199: [Python][CI][Spark] Update spark versions used on our nightly tests raulcd/arrow
3 participants
@raulcd @kiszk @assignUser