Skip to content

Commit

Permalink
[SPARK-35990][BUILD] Remove avro-sbt plugin dependency
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR removes sbt-avro plugin dependency.
In the current master, Build with SBT depends on the plugin but it seems never used.
Originally, the plugin was introduced for `flume-sink` in SPARK-1729 (#807) but `flume-sink` is no longer in Spark repository.

After SBT was upgraded to 1.x in SPARK-21708 (#29286), `avroGenerate` part was introduced in `object SQL` in `SparkBuild.scala`.
It's confusable but I understand `Test / avroGenerate := (Compile / avroGenerate).value` is for suppressing sbt-avro for `sql` sub-module.
In fact, Test/compile will fail if `Test / avroGenerate :=(Compile / avroGenerate).value` is commented out.

`sql` sub-module contains `parquet-compat.avpr` and `parquet-compat.avdl` but according to `sql/core/src/test/README.md`, they are intended to be handled by `gen-avro.sh`.

Also, in terms of Maven build, there seems to be no definition to handle `*.avpr` or `*.avdl`.

Based on the above, I think we can remove `sbt-avro`.

### Why are the changes needed?

If `sbt-avro` is really no longer used, it's confusable that `sbt-avro` related configurations are in `SparkBuild.scala`.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

GA.

Closes #33190 from sarutak/remove-avro-from-sbt.

Authored-by: Kousuke Saruta <sarutak@oss.nttdata.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
sarutak authored and dongjoon-hyun committed Jul 2, 2021
1 parent a6e00ee commit 6c4616b
Show file tree
Hide file tree
Showing 2 changed files with 1 addition and 9 deletions.
7 changes: 1 addition & 6 deletions project/SparkBuild.scala
Expand Up @@ -737,9 +737,6 @@ object Catalyst {
}

object SQL {

import sbtavro.SbtAvro.autoImport._

lazy val settings = Seq(
(console / initialCommands) :=
"""
Expand All @@ -761,10 +758,8 @@ object SQL {
|import sqlContext.implicits._
|import sqlContext._
""".stripMargin,
(console / cleanupCommands) := "sc.stop()",
Test / avroGenerate := (Compile / avroGenerate).value
(console / cleanupCommands) := "sc.stop()"
)

}

object Hive {
Expand Down
3 changes: 0 additions & 3 deletions project/plugins.sbt
Expand Up @@ -33,9 +33,6 @@ addSbtPlugin("com.typesafe" % "sbt-mima-plugin" % "0.9.2")

addSbtPlugin("com.eed3si9n" % "sbt-unidoc" % "0.4.3")

addSbtPlugin("com.cavorite" % "sbt-avro" % "2.1.1")
libraryDependencies += "org.apache.avro" % "avro-compiler" % "1.10.2"

addSbtPlugin("io.spray" % "sbt-revolver" % "0.9.1")

libraryDependencies += "org.ow2.asm" % "asm" % "9.1"
Expand Down

0 comments on commit 6c4616b

Please sign in to comment.