Using mill inside a sandbox / deterministic package manager

[lucc]

I am using NixOS and the Nix package manager. They enforce deterministic builds by building packages in a sandbox without internet connection.

Is it possible to use mill in an environment without internet? If I can figure out what needs to be downloaded and where to beforehand I can tell the package manager to fetch stuff outside of the sandbox. The nix package manager just wants to track all inputs for the build itself.

[lefou]

You can prepare your build with mill __.prepareOffline. Depending on your build and the used modules/plugins, this should be enough to fetch all online-dependencies.

[lucc]

That looks like mill will fetch stuff into the out folder. I would be more interested in understanding what I have to fetch from where myself and where to put it so that mill will find it. Or is this kind of info not exposed?

Otherwise I would have to execute the prepareOffline target on my local machine and then send the out folder together with my sources to the build sandbox. I would prefer to tell the nix daemon what has to be fetched and where to put it in the sandbox. It can then cache these things for me and I only need to send my own sources to the build daemon.

[ghostbuster91]

I would like to bring back this topic. Having a builtin solution in form of a lockfile (#120) would be great. In our project we spent a lot of time fighting against sbt. Soon we will be developing a new component and we have been debating whether use mill this time. But the lack of such support is a deal breaker for us, as we don't want to go trough the same process with mill that we went with sbt.

[lefou]

A lockfile mechanism should be rather simple to integrate into Mill. It needs to hook into the various resolvedXxx targets, like resolvedIvyDeps and resolveRunIvyDeps and just return the locked outcome. I'm very open to include it into Mill. For starters, I think it can be implemented in a custom module extending ScalaModule without any deeper integration into Mill.

A good lockfile implementation for coursier is another story. I haven't done any research into it, but given a feature set to agree on, it should be straight forward. Issue #120 already has plenty of links to start research into that direction. But the devil is in the details.

@ghostbuster91 Can you outline what your sbt solution looks like? If possible without any sbt details, just the what is locked and how it is made permanent.

[lefou]

There is probably a huge potential for misunderstanding in this topic. So we should also make clear, what we want. E.g. in coursier there is resolution (determine what artifact and versions should be in) and fetch/download (get those artifacts on the local machine), whereas in Mill both actions are currently done in the same task. If you think there is need to have each step separated, now is the chance, as we are still in the open development window for Mill 0.11.

[ghostbuster91]

There is probably a huge potential for misunderstanding in this topic.

Hopefully my explanation will decrease that risk.

Short answer is that we start sbt and then index all the files that it downloaded and create a lock file from that.

1. start sbt in a way that it fetches all the dependencies and the compiler bridge but without building the whole project (we want that step to be fast)
2. using some bash tools iterate over target directories and collect all the necessary information(jar name, pom, sha and xml)

When we want to build the project we fetch all the dependencies from the lock file before calling sbt. We override mavenRepository property to point to the local host so it won't try to download anything.

The first step was tricky because it wasn't obvious how to do that but @nrdxp found a way.
It all has been discussed in more details here: zaninime/sbt-derivation#5 (comment)

here is a like to the final script: https://gist.github.com/nrdxp/33e22cfb7d9afdef5e6c5b909bc6834d

If you think there is need to have each step separated, now is the chance, as we are still in the open development window for Mill 0.11.

As far as I understand that would be useful.

[lefou]

I suggest to start with extending ScalaModule and look for places where CoursierModule.resolveDeps task is used. If just knowing and persisting the resolution and download outcome in form of JARs is not enough, you could dig deeper in the implementation of resolveDeps, which ends in mill.modules.Jvm.resolveDependencies and mill.module.Jvm.resolveDependenciesMetadata. I guess this depends on whether you want to reuse some existing lock file format or invent your own.

[phfroidmont]

In case it helps anyone, I managed to make this work by caching manually some deps using coursier (for a Scala 3 project):

mill __.prepareOffline
mill show webapp.scalaJSToolsClasspath
cs fetch org.scalameta:scalafmt-core_2.13:3.0.0
cs fetch org.scala-lang:scala-reflect:2.13.8
cs fetch com.github.liancheng:organize-imports_2.12:0.6.0
cs fetch org.scala-lang:scala3-compiler_3:3.1.3
cs fetch org.scala-lang:scala3-sbt-bridge:3.1.3
cs fetch com.lihaoyi:mill-scalalib_2.13:0.10.5
cs fetch org.scala-js:scalajs-test-bridge_2.13:1.10.1
cs fetch org.scala-js:scalajs-library_2.13:1.7.1

Using mill __.prepareOffline all would prevent a few manual downloads but it currently tries to pull a non-existing dependency with Scala 3.

Here is a full Nix Flake example including dev env and packaging of backend and frontend webapp:

{
  description = "myapp";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs }:

    let
      pkgs = nixpkgs.legacyPackages.x86_64-linux;

      millWithJdk11 = pkgs.mill.override { jre = pkgs.jdk11; };
      depsDir = ".cache";

      millDeps = pkgs.stdenv.mkDerivation {
        name = "coursier-cache";

        nativeBuildInputs = [ millWithJdk11 pkgs.coursier ];

        outputHash = "sha256-bhMch0UyqHegXQEfgguGr50z/5QobnNWX5/tGdYeaCk=";
        outputHashAlgo = "sha256";
        outputHashMode = "recursive";

        COURSIER_CACHE = "${depsDir}/coursier";

        src = ./.;

        buildPhase = ''
          runHook preBuild
          mill __.prepareOffline
          mill show webapp.scalaJSToolsClasspath
          cs fetch org.scalameta:scalafmt-core_2.13:3.0.0
          cs fetch org.scala-lang:scala-reflect:2.13.8
          cs fetch com.github.liancheng:organize-imports_2.12:0.6.0
          cs fetch org.scala-lang:scala3-compiler_3:3.1.3
          cs fetch org.scala-lang:scala3-sbt-bridge:3.1.3
          cs fetch com.lihaoyi:mill-scalalib_2.13:0.10.5
          cs fetch org.scala-js:scalajs-test-bridge_2.13:1.10.1
          cs fetch org.scala-js:scalajs-library_2.13:1.7.1
          echo "stripping out comments containing dates"
          find ${depsDir} -name '*.properties' -type f -exec sed -i '/^#/d' {} \;
          echo "removing non-reproducible accessory files"
          find ${depsDir} -name '*.lock' -type f -print0 | xargs -r0 rm -rfv
          find ${depsDir} -name '*.log' -type f -print0 | xargs -r0 rm -rfv
          # echo "removing runtime jar"
          # find ${depsDir} -name rt.jar -delete
          echo "removing empty directories"
          find ${depsDir} -type d -empty -delete
          runHook postBuild
        '';

        installPhase = ''
          runHook preInstall
          mkdir -p $out
          cp -r ${depsDir}/coursier $out
          runHook postInstall
        '';
      };

      yarnDeps = pkgs.fetchYarnDeps {
        yarnLock = ./yarn.lock;
        sha256 = "sha256-uL3e44tIkeEOWgWczBqRFUroz39CWDfdQfrepVDhie8=";
      };
    in
    {
      devShells.x86_64-linux.default = pkgs.mkShell {
        buildInputs = with pkgs; [
          nodejs
          yarn
          millWithJdk11

          curl
          cacert

          postgresql
        ];
        shellHook = ''
          set -a
          JAVA_HOME=${pkgs.jdk11}
          set +a
        '';
      };
      packages.x86_64-linux.default = self.packages.x86_64-linux.myapp;
      packages.x86_64-linux.myapp = pkgs.stdenv.mkDerivation {
        pname = "myapp";
        version = "0.0.1";

        nativeBuildInputs = with pkgs; [
          nodejs
          yarn
          millWithJdk11
        ];

        COURSIER_CACHE = "${depsDir}/coursier";

        postConfigure = ''
          mkdir -p ${depsDir}
          cp -r ${millDeps}/coursier ${depsDir}
          chmod -R +rwX ${depsDir}
        '';

        src = ./.;

        buildPhase = ''
          # Yarn writes temporary files to $HOME.
          export HOME=$NIX_BUILD_TOP/yarn_home

          # Make yarn install packages from our offline cache, not the registry
          yarn config --offline set yarn-offline-mirror ${yarnDeps}

          # Fixup "resolved"-entries in yarn.lock to match our offline cache
          ${pkgs.fixup_yarn_lock}/bin/fixup_yarn_lock yarn.lock

          yarn install --offline --frozen-lockfile --ignore-scripts --no-progress --non-interactive

          patchShebangs node_modules

          mill mill.scalalib.scalafmt.ScalafmtModule/checkFormatAll __.sources
          mill __.fix --check
          mill all all _.test backend.assembly webapp.fullLinkJS
          yarn build:prod --offline
        '';

        installPhase = ''
          runHook preInstall
          mkdir -p $out
          cp out/backend/assembly.dest/out.jar $out/backend.jar
          cp -r dist $out/webapp
          runHook postInstall
        '';
      };
    }
  ;
}

Credit due to sbt-derivation for understanding how to prefetch the build dependencies.
A few things are specific to my build but it should be easy enough to adapt to any setup.

[vic]

This is awesome! thanks for sharing, will try to adapt for a project of mine.