sbt assembly fails

[gfl]

sbt assembly fails with new project with no other dependencies.

Expected behavior

sbt assembly should create a fat jar.

Actual behavior

It fails to create a fat jar..

Steps to reproduce the behavior

We created an empty sbt project with the following build.sbt:

name := "tsdeptest"

version := "1.0"

scalaVersion := "2.11.8"


resolvers ++= Seq(
  "twttr" at "https://maven.twttr.com/"
)

libraryDependencies ++= {
  val twitterServerVersion = "1.26.0"
  Seq(
    "com.twitter" %% "twitter-server" % twitterServerVersion
  )
}

val meta = """META.INF(.)*""".r

assemblyMergeStrategy in assembly := {
  case meta(_) => MergeStrategy.discard
  case x =>
    val oldStrategy = (assemblyMergeStrategy in assembly).value
    oldStrategy(x)
}

This in plugins.sbt:

logLevel := Level.Warn

resolvers += Resolver.sonatypeRepo("public")

addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.1.1")

addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.8.2")

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")

Running sbt clean assembly fails with the following:

[error] 1 error was encountered during merge
java.lang.RuntimeException: deduplicate: different file contents found in the following:
/Users/fernandezlopezg/.ivy2/cache/com.twitter/twitter-server_2.11/jars/twitter-server_2.11-1.25.0.jar:BUILD
/Users/fernandezlopezg/.ivy2/cache/com.twitter/finagle-http_2.11/jars/finagle-http_2.11-6.40.0.jar:BUILD
/Users/fernandezlopezg/.ivy2/cache/com.twitter/finagle-netty4_2.11/jars/finagle-netty4_2.11-6.40.0.jar:BUILD
/Users/fernandezlopezg/.ivy2/cache/com.twitter/finagle-thrift_2.11/jars/finagle-thrift_2.11-6.40.0.jar:BUILD
	at sbtassembly.Assembly$.applyStrategies(Assembly.scala:140)
	at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25)
	at sbtassembly.Assembly$.x$1$1(Assembly.scala:23)
	at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23)
	at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23)
	at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:67)
	at sbtassembly.Assembly$.inputs$1(Assembly.scala:57)
	at sbtassembly.Assembly$.apply(Assembly.scala:83)
	at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:240)
	at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:237)
	at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
	at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
	at sbt.std.Transform$$anon$4.work(System.scala:63)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
	at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
	at sbt.Execute.work(Execute.scala:235)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
	at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /Users/fernandezlopezg/.ivy2/cache/com.twitter/twitter-server_2.11/jars/twitter-server_2.11-1.25.0.jar:BUILD
[error] /Users/fernandezlopezg/.ivy2/cache/com.twitter/finagle-http_2.11/jars/finagle-http_2.11-6.40.0.jar:BUILD
[error] /Users/fernandezlopezg/.ivy2/cache/com.twitter/finagle-netty4_2.11/jars/finagle-netty4_2.11-6.40.0.jar:BUILD
[error] /Users/fernandezlopezg/.ivy2/cache/com.twitter/finagle-thrift_2.11/jars/finagle-thrift_2.11-6.40.0.jar:BUILD
[error] Total time: 30 s, completed 06-Jan-2017 10:45:10

[gfl]

It appears the issue manifests in 1.25 and 1.26.
1.24 or older doesn't have this problem.

[cacoco]

@gfl You need a merge strategy that accounts for the BUILD files unfortunately. I'm not sure how this worked previously for you (will take a look to see if something has regressed on this end) but you should update the merge strategy to be along the lines of:

  assemblyMergeStrategy in assembly := {
    case "BUILD" => MergeStrategy.discard
    case meta(_)  => MergeStrategy.last // or MergeStrategy.discard, your choice
    case other => MergeStrategy.defaultMergeStrategy(other)
  }

Additionally, you should no longer need the "https://maven.twttr.com/" resolver.

[gfl]

Thanks @cacoco, adding the BUILD files to my Merge Strategy fixed the assembly issue.

We'd already realised that the twitter resolver was not needed anymore. Thanks!

[liopei19nn]

@cacoco We came across the same issue in creating a fat jar. I am confused what are actually "BUILD" files meaning in this jar creating process?

[cacoco]

@liopei19nn BUILD files are for the build system we use internally: Pants. They are present in the source but are not used and are merely treated as text files. Since they appear in multiple jars with the same name, SBT needs to know how to "merge" them when creating a fat jar. Since they are not necessary, you can simply tell SBT to discard them.