Shapeless with SBT Assembly inside Docker

If your application:

• has a dependency on shapeless
• directly or transitively via e.g. Slick or Spray/Akka-Http
• it is built with SBT Assembly (or similar)
• and is assembled inside Docker

then this is a work around a file name length limit issue you will encounter.

• https://github.com/milessabin/shapeless/pull/338
• https://github.com/sbt/sbt-assembly/issues/69
• https://github.com/docker/docker/issues/1413

The problem

Most file systems have a file name length limit of 255.

• https://en.wikipedia.org/wiki/Filename#Length_restrictions
• https://en.wikipedia.org/wiki/Comparison_of_file_systems#Limits

Scala also hard codes its class name length to 255

• https://github.com/scala/scala/blob/2.11.x/src/reflect/scala/reflect/runtime/Settings.scala#L52

However some file systems, for example encrypted disk or virtual machines have a much shorter limit. And inside Docker the limit is 242 characters.

This is not a problem for normal use of Scala dependencies which are binaries. However this is a problem if you assemble your application JAR inside Docker using SBT Assembly. SBT Assembly explodes all dependency jars, so the classes become local files and rolls them into one application super jar.

And as shapeless can include files with a file name up to the limit of 255 characters these files will conflict and fail the assembly build when done inside Docker.

Changing this by default for everyone to e.g. 242 in shapeless is not currently advisable. As that will still break for encrypted disks, is inconsistent with the other libraries, and more importantly can loose debugging information.

• https://github.com/milessabin/shapeless/pull/338
• https://groups.google.com/forum/#!searchin/scala-internals/Xmax-classfile-name/scala-internals/hNWuwWBJCOg/GwnqXxjnK58J

Work around

• Clone https://github.com/milessabin/shapeless

• Locate branch/version of interest to you

• Add (and commit) "-Xmax-classfile-name 240" to scalacOption in Build.scala

• Similar to https://github.com/flurdy/shapeless/commit/db06cecb3a43ddfde69eb56a1746c991b50cc655

• In your application exclude the real shapeless from your build (v1.2.4 in example below)

  assemblyExcludedJars in assembly := {
val cp = (fullClasspath in assembly).value
cp filter {_.data.getName == "shapeless_2.11-1.2.4.jar"}
}

• Similar to https://github.com/flurdy/docker-compose-machine-swarm-cloud-example/blob/master/service/build.sbt#L42

Internal Maven/Ivy repository

• Change the cloned project's name or custom version in Build.scala. E.g. shapeless-docker
• https://github.com/milessabin/shapeless/blob/master/project/Build.scala#L55
• Build and deploy your shapeless clone to your internal Maven/Ivy repository
• Add this custom version to your application dependencies

Bundle JAR inside application

• Build your shapeless clone
• Add the JAR to your application's /lib folder (create the folder if not in the root)
• Similar to https://github.com/flurdy/docker-compose-machine-swarm-cloud-example/tree/master/service/lib
• Add "-Xmax-classfile-name 240" to scalacOption in your application too
• Similar to https://github.com/flurdy/docker-compose-machine-swarm-cloud-example/blob/master/service/build.sbt#L7
• Or add scalacOptions ++= Seq("-Xmax-classfile-name","240") to your ~/.sbt/0.13/local.sbt
• Similar to https://github.com/flurdy/docker-compose-machine-swarm-cloud-example/blob/master/service/src/main/resources/local.sbt