[HUDI-1869] Upgrading Spark3 To 3.1#3580
[HUDI-1869] Upgrading Spark3 To 3.1#3580pengzhiwei2018 wants to merge 1 commit intoapache:masterfrom
Conversation
this is just crazy how badly it breaks across minor versions |
If we want to both support spark 3.0 and spark 3.1, maybe way need another build config to specify the spark version. |
pengzhiwei2018
force-pushed
the
dev_spark3.1
branch
from
030cffd to
cb2e9d5
Compare
pengzhiwei2018
force-pushed
the
dev_spark3.1
branch
from
cb5cfb3 to
b418e2b
Compare
Hi @vinothchandar , I have support both spark3.0 and spark3.1. Please take a review again~ |
| <flink.version>1.13.1</flink.version> | ||
| <spark2.version>2.4.4</spark2.version> | ||
| <spark3.version>3.0.0</spark3.version> | ||
| <spark3.version>3.1.0</spark3.version> |
There was a problem hiding this comment.
Spark 3.1.0 is broken, the minimal 3.1.x version is 3.1.1 https://spark.apache.org/news/next-official-release-spark-3.1.1.html
There was a problem hiding this comment.
BTW, any reason not to use the latest patch version?
There was a problem hiding this comment.
there is a 3.1.2 now as well?
There was a problem hiding this comment.
I would suggest always use the latest patched version, for this case, Spark 2.4.8, 3.0.3, 3.1.2
|
Will review this |
| <profile> | ||
| <id>isSpark3.0</id> | ||
| <!-- for spark 3.0.x we need override the follow propeprties to package and run test--> | ||
| <properties> | ||
| <spark3.version>3.0.0</spark3.version> | ||
| <spark.version>${spark3.version}</spark.version> | ||
| <scalatest.version>3.0.1</scalatest.version> | ||
| </properties> | ||
| </profile> |
There was a problem hiding this comment.
instead of a boolean flag, better be explicit about spark version and have a maven profile match to it. we can name the profile -Dspark3.0.x just for 3.0.x and keep -Dspark3 points to the latest supported Spark 3 release.
|
@pengzhiwei2018 do you want to upgrade us to Spark 3.2 with Scala 2.13 ? |
|
This change continues in #3844 |
6 participants
What is the purpose of the pull request
Upgrading spark3 version from the spark3.0.x to spark 3.1.x
Support both spark 3.0.x and spark 3.1.x.
Build package for spark 3.0.x use:
For other spark3 version :
Brief change log
(for example:)
Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)
Committer checklist
Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.