feature request: add resource merging capability

[rkuhn]

Our shiny new config library relies upon reading all resources named reference.conf from the class path in order to obtain and merge the default values for all libraries using it (e.g. Akka, Play framework). sbt-assembly will include only the last such fragment from the class path into the fat jar, thereby losing all other defaults and making e.g. Akka crash upon initialization. The solution is to just concatenate those files (which can be done manually for example in a simple REPL session) and add that to the JAR, but sbt-assembly will not allow that because of a duplicate entry. Ignoring that resource will leave it out completely, so that it needs to be added manually afterwards (e.g. with zip in the shell).

A possible solution, which might also benefit other purposes, is the addition of merge capabilities to sbt-assembly. The build settings would include a transformation function Seq[File] -> Array[Byte] attached to a resource name, and then all resources with that name would be collected, transformed and the result added to the JAR.

[eed3si9n]

I am reluctant to add any sort of file transformation magic. I think sbt-assembly should focus on getting files together in one jar. You could probably define a sbt task to merge files together and generate one yourself.

[rkuhn]

Yes, I understand your reluctance, but adding an external merging task is not enough: the result would still need to be inserted into the fat jar, and that is currently not possible, given that it must have the same name as the things it was merged from.

[eed3si9n]

I wonder if I could refactor the sbt-assembly to give you a task that generates the raw list of files right before it gets packaged into the jar. Suppose we call it assembledFiles or something. That'll give you a hook point to place whatever you want in there after ignoring reference.conf from all jars.

[rkuhn]

That sounds like it might just cut it: then we just need a task which can for a given project retrieve and merge all reference.conf and put the result somewhere which is also ignored.

[eed3si9n]

Added assembledMappings task and published 0.7.4: a2779b1

[rkuhn]

Thanks a bunch, will look into in when I’m back from vacation (next week)!

[rkuhn]

Sorry, it took a bit longer. I got the desired effect using the following:

assembledMappings in assembly <<= (assembledMappings in assembly) map { (old: File => Seq[(File, String)]) =>
  (f: File) => old(f) :+ (file("src/main/resources/reference.conf"), "reference.conf")
}

I guess there is some nicer way with some other SBT operator, but this also works :-)

One little thing: where did you publish 0.7.4? I had to publish-local to test this …

[eed3si9n]

It's published on the community repo:

resolvers += Resolver.url("sbt-plugin-releases",
  new URL("http://scalasbt.artifactoryonline.com/scalasbt/sbt-plugin-releases/"))(Resolver.ivyStylePatterns)

[rkuhn]

Ah, okay. Plans for pushing that to central?

For the record, this is the nicest build.sbt code I could come up with:

excludedFiles in assembly <<= (excludedFiles in assembly) {
  (old) => (bases) => old(bases) ++ (bases flatMap (base => (base / "reference.conf").get))
}

assembledMappings in assembly <<= (assembledMappings in assembly) map {
  (old) => (f) => old(f) :+ (file("src/main/resources/reference.conf"), "reference.conf")
}

Going to write some actual merging to produce our reference.conf next.

[eed3si9n]

Plans for pushing that to central?

I don't think so. The next version of sbt would have it resolved by default and it's only sbt that needs it.