Custom options for XZ Tarball not working

[dEajL3kA]

Expected behaviour

Accoridng to manual it should be possible to set the XZ options to be used by packageXzTarball via:

universalArchiveOptions in (Universal, packageXzTarball)

So I added this to my "build.sbt" file:

universalArchiveOptions in (Universal, packageXzTarball) := Seq("-3")

Expected behavior:

Intead of the packageXzTarball default XZ options, which are -9e (extremely slow!), it should now be using option -3.

Actual behaviour

It is easy to verify, e.g. via htop, that packageXzTarball still is using xz -9e, rather than the desired xz -3 😞

This completey locks up my system due to immense memory usage!

It appears that option universalArchiveOptions in (Universal, packageXzTarball) is ignored or documentation is wrong.

Or am I doing something wrong?

Information

• universal:packageXzTarball
• 1.8.0
• Linux (Ubuntu 20.04)
• XZ
• xz (XZ Utils) 5.2.5
• Linux

[muuki88]

Hi @dEajL3kA

Thanks for your issue. I can't reproduce this with simple build.sbt

enablePlugins(JavaAppPackaging)

Universal / packageXzTarball / universalArchiveOptions := Seq("-3")

The command line says:

Running with tar -3 /tmp/sbt_c1d3082d/my-app-0.1.0-SNAPSHOT.tar my-app-0.1.0-SNAPSHOT

Can you share an example project that fails for you?

[dEajL3kA]

I am working on a project based on Play Framework that uses SBT as build system.

In the build.sbt file we have:

universalArchiveOptions in (Universal, packageXzTarball) := Seq("-3")

Running ./sbt clean universal:packageXzTarball results and the .tar.xz package to be created. But it easy to see, e.g. via htop, that xz gets invoked with compression parameters -9e. Apperently the custom options are not used at all.

Note that xz with option -9e, the default used by sbt-native-packager, is extremly slow and use insane amount of memory!

(The build process failed for me several times because the OOM killer ended up killing my processes)

---

Please see here:

[muuki88]

I looked closer at the code and I wasn't aware the the "archive" and "compression" phase are separated.

The universalArchiveOptions are used for the tar command, but the following xz command is not configurable.

sbt-native-packager/src/main/scala/com/typesafe/sbt/packager/universal/Archives.scala

Line 274 in 6583d70

val cmd = Seq("tar") ++ options ++ Seq(temporaryTarFile.getAbsolutePath) ++ distdirs

sbt-native-packager/src/main/scala/com/typesafe/sbt/packager/universal/Archives.scala

Line 281 in 6583d70

IO.copyFile(compressor(temporaryTarFile), tarball)

So what we actually need is a

  val universalCompressorOptions =
    SettingKey[Seq[String]]("universal-compressor-options", "Options passed to the compression command. Scope by task")

So that a user can configure

Universal / universalCompressionOptions / packageXzTarball := Seq("-3")

[dEajL3kA]

The universalArchiveOptions are used for the tar command, but the following xz command is not configurable.

I see. Makes a lot of sense why setting the xz options was not working then 😏

(This wasn't clear to me from the docs. Only by looking at the source codes I found that xz compressor options are hard-coded)

So what we actually need is a

val universalCompressorOptions =
 SettingKey[Seq[String]]("universal-compressor-options", "Options passed to the compression command. Scope by task")

So that a user can configure

Universal / universalCompressionOptions / packageXzTarball := Seq("-3")

Yeah, that would be much appreciated 🥳

(Even with this solution, I suggest changing the default from -9e to just -6, but definitely -e should be stripped)

Best regards.

[muuki88]

Release 1.9.2 is on its way

[dEajL3kA]

Release 1.9.2 is on its way

Thanks! But how do I actually use that version in my project?
https://stackoverflow.com/questions/68346952/how-to-use-sbt-native-packager-plug-in-v1-9-2-and-fix-not-found-error

[SethTisue]

1.9.2 isn't published for sbt 0.13. You'll need to upgrade to sbt 1.x. (1.5.5 just came out yesterday.)

[dEajL3kA]

1.9.2 isn't published for sbt 0.13. You'll need to upgrade to sbt 1.x. (1.5.5 just came out yesterday.)

Thanks you! I have now updated to sbt 1.x and to sbt-native-packager version 1.9.2.

Now it builds fine, but I still see the old behavior:

What am I missing? 😕

[dEajL3kA]

Never mind, after migrating to Play Framework 2.8.8 the sbt-native-packager v1.9.2 is working as expected now 🥳

(Still don't know why previous versions of Play Framework apprently used the old version, even though sbt-native-packager v1.9.2 was explicitely set in the plugins.sbt file. But, hopefully, I don't need to care anymore 😏)