Support building Graal native images in docker

[jroper]

Closes #1250

This provides support for building Graal native images in a docker container.

The simplest way to enable the docker container build is to set:

graalVMNativeImageGraalVersion := Some("19.1.1")

When setting this, the plugin will generate a docker image based on oracle/graalvm-ce:19.1.1 which installs the native-image command. It will then run this image to build the Graal native image. Options are provided to customise which base image the generated image uses, along with allowing it to use a prepackaged image rather than generating one.

The graal plugin has been restructured a little to be more idiomatic for sbt. I also introduced a dependency on the JavaAppPackaging plugin, because a graal vm image in this case is a java app, and this allows us to use the scriptClasspathOrdering, so configuring it and the behaviour of the classpath is consistent with the rest of native packager.

I also moved the setting of the dockerExecCommand setting from the DockerPlugin to the UniversalPlugin, this means that any native packager plugin that depends on UniversalPlugin can use that setting without having to depend on DockerPlugin.

[nigredo-tori]

@jroper, as you've said in #1250, there's a benefit in making the docker/native split feature somewhat reusable. I agree that there's not much hope here for reusing implementation. However, we should strive to at least make the API predictable.
To that end, I propose we add a key like this to UniversalPlugin, or some other place where it will be readily available for any SNP plugin:

val sourceDockerImage = taskKey[Option[String]]("Coordinates for a docker image to use in a command")

This is intended to specify an image for use in an arbitrary command. E.g. in the case of the GraalVMNativeImagePlugin, we would have

GraalVMNativeImage / packageBin / sourceDockerImage := {
  // build an image, or provide coordinates for an existing one
}
GraalVMNativeImage / packageBin := {
  val sourceDockerImage0 := (GraalVMNativeImage / packageBin / sourceDockerImage).value
  // ...
  sourceDockerImage0 match {
    case None =>
      buildLocal(...)
    case Some(image) =>
      buildInDockerContainer(... , image, ...)
  }
}

This way we can provide a consistent API between plugins.

Of course, there's room for improvement. One thing that comes to mind is that there might be commands that only support Docker execution, in which case a None in sourceDockerImage would make no sense. So we can make sourceDockerImage a TaskKey[String], and choose the execution branch depending on whether it is defined for the command is question.

[jroper]

and choose the execution branch depending on whether it is defined for the command is question

I'm not a big fan of logic based on whether something's defined or not, it's a bit magic. Option makes it clear. I think it's fine to just fail if it's not defined, we currently do that for mainClass, which is an Option, but is required.

[jroper]

Updated. I called the setting containerBuildImage, which reflects that this is the image used when the build is done in a container.

[jroper]

Something I've encountered while using this in a real project, I want to limit the amount of memory that native-image uses. By default it uses 80% of available physical memory on the machine (see here), which means if sbt is using more than 20%, (or if you have browsers/slack etc running) and you don't have swap, it's going to be OOM killed on Linux at least. This bug makes it impossible to limit it. So, what I'd like to do is use Docker to limit it, eg by using docker run -m4g. To do that, I need to be able to customise the docker run command used. So, how should that be allowed? Should we add a new key to the Docker plugin for the run command? Or to the Universal plugin?

[jroper]

Using this on a real project, I'm starting to think going with depending on JavaAppPackaging and using scriptClasspathOrdering is not the best idea. The Graal classpath may have a lot of dependencies that are only needed by the compiler, not needed at runtime (dependencies that contain Graal configuration). These can be marked as provided if you use the Compile classpath, but scriptClasspathOrdering uses the Runtime classpath. So I think I'll revert to just using the Compile classpath.

[muuki88]

Sorry for my late reply.

Thanks for tackling this issue 🤗 The general idea looks good to me. IMHO making this reusable shouldn't be a big focus. RPM and Debian are stable since years with rather less complaints/issues. On the other hand docker changes rather frequently and it's hard to keep up with those changes. Long story short: If there's a user willing to contribute a "docker-rpm build", then a shared API as @nigredo-tori suggested is the maximum we should do.

So, what I'd like to do is use Docker to limit it, eg by using docker run -m4g. To do that, I need to be able to customise the docker run command used. So, how should that be allowed? Should we add a new key to the Docker plugin for the run command? Or to the Universal plugin?

We already have dockerBuildOptions, but those are getting appended after the build parameter. The dockerExecCommand is already a sequence, which would allow us to hijack that, which maybe is fine for an initial start.

If this seems to error prone we could introduce something like dockerExecOptions: Setting[Seq[String]]. WDYT?

[muuki88]

I retriggered all builds as there seem to be major issues on all CI vendors -.-

[muuki88]

Thanks @jroper for the implementation ❤️
I triggered a release for 1.4.0. Should be available soon 😄

[jtjeferreira]

Using this on a real project, I'm starting to think going with depending on JavaAppPackaging and using scriptClasspathOrdering is not the best idea. The Graal classpath may have a lot of dependencies that are only needed by the compiler, not needed at runtime (dependencies that contain Graal configuration). These can be marked as provided if you use the Compile classpath, but scriptClasspathOrdering uses the Runtime classpath. So I think I'll revert to just using the Compile classpath.

@jroper this was not done, i.e use Compile classpath instead of scriptClasspathOrdering, right? When upgrading to 1.4.1 this broke my native-image-compilation because I was using provided dependencies...

[jroper]

Yeah, I should have backed out that change (I did in my local copy of the plugin).

[muuki88]

@jtjeferreira or @jroper would you like to open a pull request to fix this?