Resources defined through mappings not copied into Docker

[mathieuleclaire]

I am trying to package a scala server app into a Docker

Expected behaviour

I am suppose to get my application and a directory with some resources in the docker image.

Actual behaviour

Only application in /opt/docker/bin and dependency jars in opt/docker/lib are copied.
The resources defined through the mappings mechanism are not copied.

Information

I use:

• sbt-native-packager 1.7.3
• sbt 1.3.10
• OS: Ubuntu 20.04
• packaging java application into a docker
• My piece of code:

lazy val application = project.in(file("application")) settings (defaultSettings) dependsOn (server) enablePlugins (JavaServerAppPackaging) settings(
  mappings in Universal ++= {
    val prefix = "target/webapp"
    Seq(
      (dependencyFile in client in Compile).value -> s"$prefix/js/connect-deps.js",
      (fullOptJS in client in Compile).value.data -> s"$prefix/js/connect.js"
    ) ++ doMapping((resourceDirectory in client in Compile).value, "target") ++ doMapping((cssFile in client in Compile).value, s"$prefix/css/")
  },
  dockerEntrypoint in Universal := Seq("/opt/docker/bin/application"),
  packageName in Universal := "openmole-connect",
  organization in Universal := "openmole",
  dockerBuildOptions += "--no-cache"
)

Note: the doMapping function fetch recursively all the resources contained in directories.

Note that my target/webapp directory seems to be properly copied in target/docker/stage/opt/docker/target/webapp but not in the docker itself.

The generated Docker instructions look like this:

[info] Sending build context to Docker daemon   72.6MB
[info] Step 1/22 : FROM openjdk:8 as stage0
[info]  ---> b190ad78b520
[info] Step 2/22 : LABEL snp-multi-stage="intermediate"
[info]  ---> Running in 9c6969995766
[info] Removing intermediate container 9c6969995766
[info]  ---> 64d07a530801
[info] Step 3/22 : LABEL snp-multi-stage-id="2c8d683f-dad5-4bc2-ab11-a6f66f96bb8a"
[info]  ---> Running in 27c629362891
[info] Removing intermediate container 27c629362891
[info]  ---> c004389b95c2
[info] Step 4/22 : WORKDIR /opt/docker
[info]  ---> Running in 9c7cd81b0f42
[info] Removing intermediate container 9c7cd81b0f42
[info]  ---> 4efac068a359
[info] Step 5/22 : COPY opt /opt
[info]  ---> 32c400dc8ff5
[info] Step 6/22 : COPY 1/opt /1/opt
[info]  ---> 6b12f0c7b56d
[info] Step 7/22 : COPY 2/opt /2/opt
[info]  ---> e6ec4c732ee5
[info] Step 8/22 : USER root
[info]  ---> Running in 675f87e3b9de
[info] Removing intermediate container 675f87e3b9de
[info]  ---> fa474725d20e
[info] Step 9/22 : RUN ["chmod", "-R", "u=rX,g=rX", "/opt/docker"]
[info]  ---> Running in 7c8acb6e2212
[info] Removing intermediate container 7c8acb6e2212
[info]  ---> 7c401cea0e70
[info] Step 10/22 : RUN ["chmod", "-R", "u=rX,g=rX", "/1/opt/docker"]
[info]  ---> Running in 06214e76114c
[info] Removing intermediate container 06214e76114c
[info]  ---> 9f56b46dec14
[info] Step 11/22 : RUN ["chmod", "-R", "u=rX,g=rX", "/2/opt/docker"]
[info]  ---> Running in d7126964057a
[info] Removing intermediate container d7126964057a
[info]  ---> 3da3628e2d99
[info] Step 12/22 : RUN ["chmod", "u+x,g+x", "/1/opt/docker/bin/application"]
[info]  ---> Running in fe5c10827ff6
[info] Removing intermediate container fe5c10827ff6
[info]  ---> 180fff21da6a
[info] Step 13/22 : FROM openjdk:8 as mainstage
[info]  ---> b190ad78b520
[info] Step 14/22 : USER root
[info]  ---> Running in 3cd3a06042d6
[info] Removing intermediate container 3cd3a06042d6
[info]  ---> 96de0ebdbe78
[info] Step 15/22 : RUN id -u demiourgos728 1>/dev/null 2>&1 || (( getent group 0 1>/dev/null 2>&1 || ( type groupadd 1>/dev/null 2>&1 && groupadd -g 0 root || addgroup -g 0 -S root )) && ( type useradd 1>/dev/null 2>&1 && useradd --system --create-home --uid 1001 --gid 0 demiourgos728 || adduser -S -u 1001 -G root demiourgos728 ))
[info]  ---> Running in 15a57b0eb978
[info] Removing intermediate container 15a57b0eb978
[info]  ---> 59a0dd3dc380
[info] Step 16/22 : WORKDIR /opt/docker
[info]  ---> Running in ddc5b63af17d
[info] Removing intermediate container ddc5b63af17d
[info]  ---> cd44f5ba5400
[info] Step 17/22 : COPY --from=stage0 --chown=demiourgos728:root /opt/docker /opt/docker
[info]  ---> 5e256b40af0e
[info] Step 18/22 : COPY --from=stage0 --chown=demiourgos728:root /1/opt/docker /opt/docker
[info]  ---> abb06e4be250
[info] Step 19/22 : COPY --from=stage0 --chown=demiourgos728:root /2/opt/docker /opt/docker
[info]  ---> 915afa5d368b
[info] Step 20/22 : USER 1001:0
[info]  ---> Running in db401963a3af
[info] Removing intermediate container db401963a3af
[info]  ---> 7823f2903569
[info] Step 21/22 : ENTRYPOINT ["/opt/docker/bin/application"]
[info]  ---> Running in ef833e26ac04
[info] Removing intermediate container ef833e26ac04
[info]  ---> 0a1e5cb53cde
[info] Step 22/22 : CMD []
[info]  ---> Running in 9eabe2698166
[info] Removing intermediate container 9eabe2698166
[info]  ---> 8a445e1cc20a
[info] Successfully built 8a445e1cc20a
[info] Successfully tagged application:0.1.0-SNAPSHOT
[info] Removing intermediate image(s) (labeled "snp-multi-stage-id=2c8d683f-dad5-4bc2-ab11-a6f66f96bb8a") 
[info] Deleted Images:
[info] deleted: sha256:180fff21da6af8d2da62373582006795770fe7ecea60b8256b7d7aa8463116c4
[info] deleted: sha256:f04603065a55aa78ace281a1af9d05e67a01a5b065c9869519805c3d3f4a5a2f
[info] deleted: sha256:3da3628e2d99ad32198bc6c679e78874ebebe84a3c1457fb1ef756b01002aa5e
[info] deleted: sha256:0a886f2e99dc51acfd17748939c9c2252e9ffe87c89e58e7cf6b868487ace60f
[info] deleted: sha256:9f56b46dec141e1210c0600e82ba1d3de1d564c2db1f0d01efac1c16d1847629
[info] deleted: sha256:e706f8706b81debe373c4e3f916bd8e97c299c9dc8487c164ef4f0b6c55af273
[info] deleted: sha256:7c401cea0e70d1a9fc2ffe9bfdb7c7ae937c47a1c648fdf38cf36b0891901fec
[info] deleted: sha256:f62020e6a514c2252f62d72956d899d18079a7377123681c4b75140d037dd1a9
[info] deleted: sha256:fa474725d20e82fbbcc726e9c67959735458f0ce7f7f2ca1894221a3b92dd649
[info] deleted: sha256:e6ec4c732ee59d5ba533636e35796564f8b5c96c0afff4a901d1a1369d74e29d
[info] deleted: sha256:a6217b39b6cdd15da3748592eb3f18a7e09778aa30cec6414fa505777972e4ee
[info] deleted: sha256:6b12f0c7b56dab95c491248ee7eeaaea58ced77e95f82d9ff5e8da6f47ae6e1b
[info] deleted: sha256:ad538d8f4a615b3c23c739f6a85d7060b2e427140d527e7576babed438932b3f
[info] deleted: sha256:32c400dc8ff5610ce1f86701ac7ddd8dfa38cc2718b781c86546aa9046bac9d2
[info] deleted: sha256:c1997359f4a8986c682a6173874b5bc98fb34b1215d3ce8473c18c4661dd8980
[info] deleted: sha256:4efac068a3596186b42f1fadec54bc920153eec51a0b0979eeab988cd1e56f28
[info] deleted: sha256:d772f376cfba48e0abc58958af608d4d7ac98f622d7be0257f49e9da94528670
[info] deleted: sha256:c004389b95c29dfefd89724448b09f77dcaa49f6da46bf0a5532ffec06613d98
[info] deleted: sha256:64d07a53080185958202592ebfd2384e80420b1b4b19ed04de6370aedcbc8e32

[muuki88]

Hi @mathieuleclaire

Thanks for the detailed issue description. Let's unwrap this 😃

Setup and intentions

I assume what you are trying to do is packaging your frontend assets (scalajs) into the resulting server so they can be served. You have split up your build in two modules server and client where serverdepends onclient. This is pretty standard stuff so far.

The actual error

I assume that the assets are not properly being served?

How to

Because the server depends on the client, the client project will be available as a jar in the lib/ directory. So you scalajs resources should be available in the client.jar and not directly in the server.

If my assumption about the actual error is correct than you are like me and a lot of other jvm developers 😂 Loading resources from jars on the classpath is somewhat error prone and painful to debug on the jvm.

Can you post the actual problem you are facing if I'm mistaken 😄

[mathieuleclaire]

Hi @muuki88,
Your are right, I try to build a server application, which serve frontend stuff compiled and gathered with scalajs. This part is OK: I can generate the js I need, pack my js dependency resources and CSSs.

What I want

It is pretty easy actually: I need in my docker both my server applicatons and jars as well as my frontend assets in the same place. Something like in /opt/docker/:

• bin/application
• lib/*.jars
• webapp/ (containing frontend directories js, css, fonts, img)

What I have

Only the first two directories (bin and lib), and not the webapp one.

How

I try copy my frontend stuff in the /opt/docker/webapp with the mappings methods. But the resources are not copied.
Note however that I can find them copied locally in the my target/docker/stage/opt/docker/target/webapp

Is the way I do is the way to do ? If yes, what is my mistake ?

[muuki88]

Thanks for the clarification @mathieuleclaire 😃

Can you show the output of

$ sbt "show server / Docker / mappings"

It should show all files that end up in the docker image. If the files don't appear there, then it's a configuration error, if they do, then there's a bug in native-packager.

Does the client module generate a jar with the js files included? I haven't used scalajs in ages. Because usually that's what I would recommend that you do less configuration in sbt and rely on the classpath. E.g. if the jar produced by the client module has the assets under webapp then you could load an asset with

// or similar
getClass.getClassLoader.getInputStream("/webapp/styles.css")

Most scala http servers have built in support for this, e.g. akka getResourceDirectory

[mathieuleclaire]

I ran show docker:mappings in my application project. I get

• all the jars
• the application executables (first 2 lines below)
• and also all the resources I want (after first 2 lines)

The thing I observe is the frontend resources are not mapped in the application directory but in the client one. Could it be the issue ?

[info] * (/home/mathieu/work/cogit/openmole-connect/application/target/universal/scripts/bin/application,/opt/docker/bin/application)
[info] * (/home/mathieu/work/cogit/openmole-connect/application/target/universal/scripts/bin/application.bat,/opt/docker/bin/application.bat)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/target/scala-2.13/client-jsdeps.min.js,/opt/docker/target/webapp/js/connect-deps.js)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/target/scala-2.13/client-opt.js,/opt/docker/target/webapp/js/connect.js)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/src/main/resources/webapp/WEB-INF,/opt/docker/target/webapp/WEB-INF)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/src/main/resources/webapp/fonts,/opt/docker/target/webapp/fonts)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/src/main/resources/webapp/css,/opt/docker/target/webapp/css)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/src/main/resources/webapp/img,/opt/docker/target/webapp/img)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/src/main/resources/webapp/WEB-INF/web.xml,/opt/docker/target/webapp/WEB-INF/web.xml)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/src/main/resources/webapp/fonts/icomoon.ttf,/opt/docker/target/webapp/fonts/icomoon.ttf)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/src/main/resources/webapp/fonts/icomoon.svg,/opt/docker/target/webapp/fonts/icomoon.svg)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/src/main/resources/webapp/fonts/icomoon.woff,/opt/docker/target/webapp/fonts/icomoon.woff)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/src/main/resources/webapp/fonts/glyphicons-halflings-regular.ttf,/opt/docker/target/webapp/fonts/glyphicons-halflings-regular.ttf)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/src/main/resources/webapp/fonts/glyphicons-halflings-regular.woff2,/opt/docker/target/webapp/fonts/glyphicons-halflings-regular.woff2)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/src/main/resources/webapp/fonts/glyphicons-halflings-regular.eot,/opt/docker/target/webapp/fonts/glyphicons-halflings-regular.eot)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/src/main/resources/webapp/fonts/gi.otf,/opt/docker/target/webapp/fonts/gi.otf)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/src/main/resources/webapp/fonts/icomoon.eot,/opt/docker/target/webapp/fonts/icomoon.eot)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/src/main/resources/webapp/fonts/glyphicons-halflings-regular.svg,/opt/docker/target/webapp/fonts/glyphicons-halflings-regular.svg)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/src/main/resources/webapp/fonts/glyphicons-halflings-regular.woff,/opt/docker/target/webapp/fonts/glyphicons-halflings-regular.woff)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/src/main/resources/webapp/css/style.css,/opt/docker/target/webapp/css/style.css)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/src/main/resources/webapp/img/logo.png,/opt/docker/target/webapp/img/logo.png)
[info] * (/home/mathieu/work/cogit/openmole-connect/client/target/css/deps.css,/opt/docker/target/webapp/css//deps.css)

[mathieuleclaire]

Well, I fixed my problem. But I am not very sure how (and I do not like this !)
I essentially changed in Universal in in Docker, but it was my very initial config, which did not work.
Thanks anyway !

Here is the working snippet for memory:

val prefix = "/opt/docker/application/target"
lazy val application = project.in(file("application")) settings (defaultSettings) dependsOn(server) enablePlugins (JavaServerAppPackaging) settings(
  mappings in Docker ++= Seq(
    (dependencyFile in client in Compile).value -> s"$prefix/webapp/js/connect-deps.js",
    (fullOptJS in client in Compile).value.data -> s"$prefix/webapp/js/connect.js"
  ) ++ doMapping((resourceDirectory in client in Compile).value, prefix)
    ++ doMapping((cssFile in client in target).value, s"$prefix/webapp/css/"),
  packageName in Docker := "openmole-connect",
  organization in Docker := "openmole"
)

[muuki88]

Thanks for sharing. This is strange. The mappings in Docker should inherit from mappings in Universal.

The docker plugin is quite complex due to the nature of docker being complex. I'll leave this open as there's
something weird going on, but I don't have the time to look deeper into it 😢