[Enhancement] Override BatchWrite#useCommitCoordinator to run on DataBricks 13.1 #79
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this:
Which issues of this PR fixes :
Fixes #
Problem Summary(Required) :
What's the problem
Running spark connector on DataBricks 13.1 (powered by Spark 3.4.0) will fail because of the below exception. The version of spark connector is 1.1.0.
How to solve it
DBR 13.1 is powered by Spark 3.4.0, but it also includes an additional improvement https://issues.apache.org/jira/browse/SPARK-42968. The improvement introduces a default method
useCommitCoordinator
in interfaceStreamingWrite
. BecauseStarRocksWrite
implements both interfaceBatchWrite
andStreamingWrite
which both have default methoduseCommitCoordinator
on DBR 13.1, so it will leads default method conflict as the exception says. We could override the method inStarRocksWrite
to solve it. see https://www.geeksforgeeks.org/resolving-conflicts-during-multiple-inheritance-in-java/.This improvement is introduced since Spark 3.5, but DBR introduce it in advance, and that's why the connector can run on community Spark 3.4, but failed on DBR 13.1. It seems DBR is not completely compatible with the community Spark.
[1] DataBrick 13.1 release note
Checklist: