[GH-3297] Drop Spark 3.4 support - #3300
Merged
Merged
Conversation
Spark 3.4 reached end of life in October 2024 with 3.4.4 as its final maintenance release, and the major cloud runtimes have since removed it. Build: - Remove the spark/spark-3.4 module and the sedona-spark-3.4 profile - Make 3.5 the default spark.version/spark.compat.version, and the profile that deploys the parent and common modules - Drop spark-3.5 from enable-all-submodules now that it is the default CI: remove the 3.4 rows from the java, python, r and example matrices. The python 3.4 row also carried the Python 3.8 and shapely 1 coverage, so that row moves to Spark 3.5 rather than being deleted. The Spark Connect step's `matrix.spark >= '3.4.0'` guard is now always true. Python: raise the pyspark floor to 3.5.0 and drop the pyspark < 3.4 skip markers from the raster tests. Scala: PartitionedFile's constructor is identical from 3.5.0 onwards, so StacPartitionReader builds one directly instead of reflectively choosing between the 7- and 8-argument forms. SchemaMergeUtils keeps its reflective fallback: StructType.merge(StructType, Boolean) does not exist in 3.5.0, only from 3.5.1. Docs: update the compatibility tables, compile and coordinate examples, and the release and snapshot build matrices, which still named 3.3. Also remove spark-version-converter.py, a Spark 2/3 helper whose target paths have not existed since the module layout changed. Closes apache#3297
… builds Two problems found in review of the Spark 3.4 removal. Spark 4.x has no Scala 2.12 build: spark-core_2.12:4.0.0 and 4.1.1 are both absent from Maven Central, and Sedona has only ever published sedona-spark-shaded-4.0_2.13. Building with -Dspark=4.0 -Dscala=2.12 fails resolving spark-core_2.12 and spark-sql_2.12, because the scala2.12 profile is declared after sedona-spark-4.0 and so wins on scala.compat.version. - Drop the Spark 4.0 / Scala 2.12 deploy step from the snapshot docs - Point the tutorial examples at sedona-spark-shaded-4.0_2.13 - Remove the Spark 4.0 / Scala 2.12 coordinate tabs, and unmark that combination in the compatibility tables. publish.md already skipped Spark 4.x with Scala 2.12 as unsupported, so the tables and tabs contradicted the release procedure. The docker docs built against Spark 3.4.1. build.sh derives -Dspark from the first two components of that version, and with the 3.4 profile gone Maven matched no profile and silently used the new 3.5 defaults, installing a 3.5 shaded jar into a Spark 3.4 image. The STAC reader now calls the eight-argument PartitionedFile constructor directly, which Spark 3.4 does not have, so that mismatch would fail at runtime. - Build the documented images against Spark 4.0.1, which is what the dockerfile already defaults to and what CI exercises - Make build.sh reject Spark versions outside 3.5, 4.0 and 4.1 rather than letting Maven fall back to its defaults
The compile tab read "Spark 3.5+ Scala 2.12" and told readers to
substitute a higher Spark version, and the EMR note said "Spark 3.5+ and
Scala 2.12". Neither holds: Spark 4.x publishes no Scala 2.12 artifacts,
so following either instruction fails.
mvn -pl spark/common -am -Dspark=4.0 -Dscala=2.12 -DskipTests compile
=> Could not resolve dependencies ... spark-core_2.12:jar:4.0.0,
spark-sql_2.12:jar:4.0.0
Scope the Scala 2.12 tab to Spark 3.5 and say why. The Scala 2.13 tab
keeps its "higher Spark versions" note, which is accurate for 3.5, 4.0
and 4.1. The EMR note now points Spark 4.0 and above at the 2.13
artifact.
james-willis
approved these changes
Aug 29, 2026
Spark 3.3 was dropped in Sedona 1.8.0, but the setup and tutorial pages still told readers to pull sedona-spark-shaded-3.3_2.12 at the current Sedona version, an artifact that has not been published since 1.7.1. Dropping 3.4 made this worse in emr.md, whose note recommended 3.5_2.12 while the script below it downloaded 3.3_2.12. Point the 28 coordinates in emr, install-scala, install-python, sql-pure-sql, sql and raster (plus their Chinese mirrors) at 3.5_2.12, and update the surrounding "replace the 3.3" prose to match. Both artifacts resolve on Maven Central at 1.9.1. sql-pure-sql also carried a "Spark 3.3+ and Scala 2.12" tab with the same open-ended range that was corrected in compile.md and emr.md: Spark 4.x has no Scala 2.12 build, so the tab is now Spark 3.5 and points at 4.0_2.13 for later versions. The EMR walkthrough ran on EMR 6.9.0, which ships Spark 3.3. Move it to EMR 7.9.0; component versions are from the AWS EMR 7.x application versions table. The remaining 3.3 references are deliberate and kept: glue.md documents the Glue 4.0 fallback for users pinned to Sedona 1.7.1, and release-notes.md records a 1.4.1 snippet.
…source
The guard added earlier ran before the published-version branch, so it
also rejected rebuilds of historical images:
docker/build.sh 3.4.1 1.9.0 local 33.5
=> Unsupported Spark version: 3.4.1...
That build is legitimate. It never compiles: install-sedona.sh
downloads sedona-spark-shaded-3.4_2.13 from Maven Central, which is
published for 1.9.0. Only the SEDONA_VERSION=latest branch runs mvn and
so can silently fall back to the default Spark profile.
Move the check, and the SEDONA_SPARK_VERSION it guards, into that
branch. SEDONA_SPARK_VERSION had no other reader.
Verified against a stubbed mvn and docker:
- 3.4.1 1.9.0, 3.3.0 1.7.1, 3.5.8 1.9.1 -> exit 0, reaches image build
- 3.4.1 latest, 3.3.0 latest -> exit 1, rejected
- 3.5.8/4.0.1/4.1.1 latest -> exit 0, compiles with -Dspark
The Spark 4 guidance sent every "4.0 and above" reader to sedona-spark-shaded-4.0_2.13, but 4.1 is supported and publishes its own sedona-spark-shaded-4.1_2.13. Keep the Scala 2.13 point and tell readers to match the artifact's Spark major.minor suffix, naming both 4.0 and 4.1. emr.md carried the same sentence from the same edit, so it is fixed too. raster.md and sql.md already said to substitute the major.minor version and only used 4.0_2.13 as an illustration, so they stand. Checked every Sedona spark coordinate named under docs/ against Maven Central. All resolve at 1.9.1 except the deliberately historical ones, which resolve at the Sedona version their page pins: sedona-spark-shaded-3.3_2.12 at 1.7.1 (glue.md's Glue 4.0 fallback), sedona-spark-shaded-3.4_2.12 at 1.5.1 and 1.6.x (the Fabric and Synapse walkthroughs), plus release-notes.md and a legacy notebook transcript.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Did you read the Contributor Guide?
Is this PR related to a ticket?
[GH-XXX] my subject. Closes Drop Spark 3.4 support #3297What changes were proposed in this PR?
Spark 3.4 reached end of life in October 2024 with 3.4.4 as its final maintenance release, and the major cloud runtimes have since removed it.
Build
spark/spark-3.4module and thesedona-spark-3.4profile.spark.version/spark.compat.version, and the profile that deploys the parent and common modules (skip.deploy.common.modules).spark-3.5fromenable-all-submodules, since it is now the default module.CI
matrix.spark >= '3.4.0'guard is now always true and has been removed.docs.ymlreadsspark.versionfrom the POM, so it follows the new default automatically.Python
>=3.5.0.pyspark.__version__ < "3.4"skip markers from the raster tests, and the imports they orphaned.Scala
PartitionedFile's constructor is identical from 3.5.0 onwards, soStacPartitionReadernow builds one directly instead of reflectively choosing between the 7- and 8-argument forms. Verified against the published jars: 3.5.0, 3.5.8, 4.0.2 and 4.1.1 all expose(InternalRow, SparkPath, long, long, String[], long, long, Map); only 3.4 had the 7-argument form. The private-constructor reflection increateSparkPathgoes with it, superseded by the publicSparkPath.fromPathString.SchemaMergeUtils.safeMergekeeps its reflective fallback.StructType.merge(StructType, Boolean)does not exist in Spark 3.5.0 — it arrives in 3.5.1 — and 3.5.0 remains supported. Only the stale comments were corrected.AbstractCatalogaggregate-builder reflection also stays; it splits Spark 3.x from 4.x and is still live for 3.5.Docs
publish.mdandsnapshot.md: the release and snapshot build matrices still named Spark 3.3, dropped back in 1.8.0, and would have been entirely invalid after this change.Also
spark-version-converter.py, a Spark 2/3 helper whose target paths (sql/src/main/scala/...) have not existed since the module layout changed. Nothing in the tree references it.How was this patch tested?
Builds, all
BUILD SUCCESS:-Dspark=3.5 -Dscala=2.12(JDK 11)-Dspark=3.5 -Dscala=2.13(JDK 11)-Dspark=4.0 -Dscala=2.13(JDK 17)-Dspark=4.1 -Dscala=2.13(JDK 17)mvn clean install -DskipTestswith no flags, confirming the default now resolves tosedona-spark-3.5_2.12Targeted tests for the changed
StacPartitionReaderpath —StacPartitionReaderTest,StacBatchTest,StacDataSourceTest— on both3.5/2.12and4.1/2.13: 32 succeeded, 0 failed on each. The 4 canceled tests are the env-gated live STAC service tests (STAC_PUBLIC_URL,STAC_AUTH_URL), canceled identically on both.pre-commit run --all-filespasses, andmvn spotless:applyproduces no changes.Did this PR include necessary documentation updates?