resolving deduplicate problems in build.sbt

[behrad]

How should I deal with

java.lang.RuntimeException: deduplicate: different file contents found in the following:
logger.xml
/home/jrad/.ivy2/cache/org.reactivecouchbase/reactivecouchbase-core_2.10/jars/reactivecouchbase-core_2.10-0.4-SNAPSHOT.jar:logger.xml

my build.sbt snippet:

assemblySettings

excludedJars in assembly <<= (fullClasspath in assembly) map { cp =>
  cp filter {x => x.data.getName.matches("jsr173_api-1.0.jar")}
}

test in assembly := {}

[eed3si9n]

See https://github.com/sbt/sbt-assembly#merge-strategy

[alivcor]

Just a friendly suggestion - why is merging such a big hassle in this plugin, seeing so many issues around this, presence of so many issues around this is probably reason enough to believe it can be done in a better way, without users having to deal with all this complexity. Something isn't right.

[daggerok]

I had similar problem with akka 2.64 and akka-serialization-jackson:

[error] (assembly) deduplicate: different file contents found in the following:
[error] /path/to/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.10.3/jackson-annotations-2.10.3.jar:module-info.class
[error] /path/to/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.10.3/jackson-core-2.10.3.jar:module-info.class
[error] /path/to/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.10.3/jackson-databind-2.10.3.jar:module-info.class
[error] /path/to/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/dataformat/jackson-dataformat-cbor/2.10.3/jackson-dataformat-cbor-2.10.3.jar:module-info.class
[error] /path/to/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/datatype/jackson-datatype-jdk8/2.10.3/jackson-datatype-jdk8-2.10.3.jar:module-info.class
[error] /path/to/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/datatype/jackson-datatype-jsr310/2.10.3/jackson-datatype-jsr310-2.10.3.jar:module-info.class
[error] /path/to/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-module-parameter-names/2.10.3/jackson-module-parameter-names-2.10.3.jar:module-info.class
[error] /path/to/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/module/jackson-module-paranamer/2.10.3/jackson-module-paranamer-2.10.3.jar:module-info.class

My problem was solved with next assembly config:

assemblyMergeStrategy in assembly := {
  case PathList("jackson-annotations-2.10.3.jar", xs @ _*) => MergeStrategy.last
  case PathList("jackson-core-2.10.3.jar", xs @ _*) => MergeStrategy.last
  case PathList("jackson-databind-2.10.3.jar", xs @ _*) => MergeStrategy.last
  case PathList("jackson-dataformat-cbor-2.10.3.jar", xs @ _*) => MergeStrategy.last
  case PathList("jackson-datatype-jdk8-2.10.3.jar", xs @ _*) => MergeStrategy.last
  case PathList("jackson-datatype-jsr310-2.10.3.jar", xs @ _*) => MergeStrategy.last
  case PathList("jackson-module-parameter-names-2.10.3.jar", xs @ _*) => MergeStrategy.last
  case PathList("jackson-module-paranamer-2.10.3.jar", xs @ _*) => MergeStrategy.last
  case _ => MergeStrategy.first
}

But I still cannot understand what I've just did and why it solves my problem...

---

Regards,
Maksim

[vrann]

In addition to what @daggerok recommended (thanks it worked!) to fix the next set of problems such as No configuration setting found for key 'akka.version', add following case to assemblyMergeStrategy:

case PathList("reference.conf") => MergeStrategy.concat

[deekayman]

This does not work.

[error] (common / assembly) deduplicate: different file contents found in the following:
[error] /Users/dave/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.12.0/jackson-annotations-2.12.0.jar:module-info.class
[error] /Users/dave/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.12.0/jackson-core-2.12.0.jar:module-info.class
[error] /Users/dave/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.12.0/jackson-databind-2.12.0.jar:module-info.class

[Shestaevd]

This does not work.

[error] (common / assembly) deduplicate: different file contents found in the following:
[error] /Users/dave/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.12.0/jackson-annotations-2.12.0.jar:module-info.class
[error] /Users/dave/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.12.0/jackson-core-2.12.0.jar:module-info.class
[error] /Users/dave/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.12.0/jackson-databind-2.12.0.jar:module-info.class

you don't need module-info.class, so you can simply discard it by
case PathList("module-info.class") => MergeStrategy.discard

[dmedser]

This does not work.

[error] (common / assembly) deduplicate: different file contents found in the following:
[error] /Users/dave/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.12.0/jackson-annotations-2.12.0.jar:module-info.class
[error] /Users/dave/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.12.0/jackson-core-2.12.0.jar:module-info.class
[error] /Users/dave/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.12.0/jackson-databind-2.12.0.jar:module-info.class

you don't need module-info.class, so you can simply discard it by
case PathList("module-info.class") => MergeStrategy.discard

Thank you very much! I had the same problem and it worked!

[hagay3]

The module-info.clas file has moved in many libraries. Here is the updated solution

ThisBuild / assemblyMergeStrategy  := {
  case PathList("module-info.class") => MergeStrategy.discard
  case x if x.endsWith("/module-info.class") => MergeStrategy.discard
  case x =>
    val oldStrategy = (ThisBuild / assemblyMergeStrategy).value
    oldStrategy(x)
}

[eed3si9n]

The point of the PathList(...) is to work around the fact that on Windows path is \, so this might be better? (I haven't checked this):

ThisBuild / assemblyMergeStrategy  := {
  case PathList(ps @ _*) if ps.last == "module-info.class" => MergeStrategy.discard
  case x =>
    val oldStrategy = (ThisBuild / assemblyMergeStrategy).value
    oldStrategy(x)
}

[fernanluyano]

It's still an issue after years. Is this fixed? I really prefer a different build plugin than going through hundreds of dedup conflicts

[coderman2022]

Still an issue. I switched to Maven Shade Plugin. Works perfect.

http://www.mkyong.com/maven/create-a-fat-jar-file-maven-shade-plugin

[nikhilsimha]

What is the most recent way to fix this?

[coderman2022]

What is the most recent way to fix this?

None that I've seen. Best fix is to move to Maven Shade Plugin.

This issue should not be Closed. This has been a bug since at least 2015.