[WIP] Support for Spark 3.0

[imback82]

What changes were proposed in this pull request?

This PR proposes to support Spark 3.0. I will share more about branching strategies, etc. as a separate PR.

Why are the changes needed?

To support Spark 3.0.

Does this PR introduce any user-facing change?

Yes. The user needs to consume this library along with Spark 3.0. This is not compatible with Scala 2.11 any longer.

How was this patch tested?

Updated existing tests.

[sezruby]

Do you have any plan for code management of both spark 2.4.6 and 3.0.0?
I think we should manage different codebases for spark2.4.6 and 3.0.0 and provide build.sbt option accordingly.

[imback82]

The current plan is to have 3.0 support in master and create a branch-2.4 for 2.4 support. The development process is to create a PR against master and merge it to both master/branch-2.4.

[sezruby]

That should be fine for developers :)

[imback82]

@sezruby in Delta 8.0, there were some changes in the API. Could you confirm if these changes are fine?

[imback82]

@andrei-ionescu In Spark 3.0, DataSourceV2Relation signature has changed and takes in Table now. Could you help fixing the it?

[error] /home/vsts/work/1/s/src/main/scala/com/microsoft/hyperspace/index/sources/iceberg/IcebergRelation.scala:58: value source is not a member of org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
[error]   override def allFiles: Seq[FileStatus] = plan.source match {
[error]                                                 ^
[error] /home/vsts/work/1/s/src/main/scala/com/microsoft/hyperspace/index/sources/iceberg/IcebergRelation.scala:66: value source is not a member of org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
[error]   override def partitionBasePath: Option[String] = plan.source match {
[error]                                                         ^
[error] /home/vsts/work/1/s/src/main/scala/com/microsoft/hyperspace/index/sources/iceberg/IcebergRelation.scala:85: value source is not a member of org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
[error]     plan.source match {
[error]          ^
[error] /home/vsts/work/1/s/src/main/scala/com/microsoft/hyperspace/index/sources/iceberg/IcebergRelation.scala:87: not found: type DataSourceOptions
[error]         val dsOpts = new DataSourceOptions(plan.options.asJava)
[error]                          ^

[imback82]

closed by #421