Skip to content

Adapt spark versions for 24.7 release #426

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
maltesander merged 4 commits into from
Jul 16, 2024
+12 −8
Merged

Adapt spark versions for 24.7 release #426

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0e967bf
add spark version 4.0.0-preview1
maltesander Jul 11, 2024
41ee49f
remove 4.0.0-preview1
maltesander Jul 11, 2024
70b5e68
fix: iceberg version when using custom images
razvan Jul 15, 2024
ce92261
Merge branch 'main' into chore/bump-spark-versions
razvan Jul 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions docs/modules/spark-k8s/partials/supported-versions.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@
// Stackable Platform documentation.
// Please sort the versions in descending order (newest first)

- 3.5.1 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 11)
- 3.5.0 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 11) (deprecated)
- 3.4.2 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 11) (LTS)
- 3.4.1 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 11) (deprecated)
- 3.5.1 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 17) (LTS)
- 3.4.3 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 11) (deprecated)
- 3.4.2 (Hadoop 3.3.4, Scala 2.12, Python 3.11, Java 11) (deprecated)
10 changes: 8 additions & 2 deletions tests/templates/kuttl/iceberg/10-deploy-spark-app.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,14 @@ spec:
mountPath: /stackable/spark/jobs
deps:
packages:
# need to extract only the major and minor versions
- org.apache.iceberg:iceberg-spark-runtime-{{ test_scenario['values']['spark'].rsplit('.', maxsplit=1)[0] }}_2.12:1.4.0
#
# The iceberg runtime contains the spark and scala versions in the form :
#
# <spark major>.<spark minor>_<scala major>.<scala minor>
#
# We extract the spark parts from the test scenario value.
#
- org.apache.iceberg:iceberg-spark-runtime-{{ ".".join(test_scenario['values']['spark'].split('.')[:2]) }}_2.12:1.5.2
volumes:
- name: script
configMap:
Expand Down
3 changes: 1 addition & 2 deletions tests/test-definition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,8 @@ dimensions:
- "false"
- name: spark
values:
- 3.4.1
- 3.4.2
- 3.5.0
- 3.4.3
- 3.5.1
# Alternatively, if you want to use a custom image, append a comma and the full image name to the product version
# as in the example below.
Expand Down