Lockfile support

[lovesegfault]

Currently the hash of the dependencies' fixed output derivation has to be manually updated whenever there is a change.

We could allow users to use sbt-dependency-lock and then derive the hash of the FOD from there.

This is very similar to what buildRustPackage does with cargoLock.lockFile.

cc. @nrdxp @jonringer

[lovesegfault]

The way this is done for Rust can be seen in pkgs/build-support/rust/import-cargo-lock.nix

[roberth]

This is also the direction which Sbtix wants to take, but it doesn't seem to be in development anymore. See https://gitlab.com/teozkr/Sbtix/-/issues/41 Sbtix in its current architecture does produce a lockfile that can be used in production projects, at least when said projects do not override dependencies in the wrong way, which can trip up sbtix and cause the build to fail.

sbt-dependency-lock could be a double-edged sword. It shows that the (non-Nix) scala community has interest in lockfiles, but its goal isn't necessarily to make builds hermetic (?). Getting all dependencies is actually somewhat hard, because of the plugin dependencies, so it'd be important to know whether sbt-dependency-lock considers those. UPDATE: I've written an issue about adding the plugins stringbean/sbt-dependency-lock#28

[lovesegfault]

A stub of how this might look:

{ fetchurl, lib, runCommand, jq }:

{
  lockFile ? null
, lockFileContents ? null
, outputHashes ? null
} @ args:

assert (lockFile == null) != (lockFileContents == null);

let
  lockFileContents =
    if lockFile != null
    then builtins.readFile lockFile
    else args.lockFileContents;

  lockData = builtints.fromJSON lockFileContents;

  fetchDependency =
    { org
    , name
    , version
    , artifacts
    , ...
    }:
    let
      orgPath = builtins.replaceStrings [ "." ] [ "/" ] org;
      # FIXME: I don't really understand what these artifacts are. The lockfile
      # reference says they're a set of the filename and hash, but the filenames
      # in the example lockfiles don't match anything available on maven
      # central. It's also weird that there's always only one artifact listed;
      # why is it a list?
      # https://stringbean.github.io/sbt-dependency-lock/file-formats/version-1.html
      sha1 = lib.removePrefix "sha1:" (builtins.head artifacts).hash;
    in
    fetchurl {
      url = "https://repo1.maven.org/maven2/${orgPath}/${name}/${version}/${name}-${version}.jar";
      inherit sha1;
    };


  dependencies = map fetchDependency lockData.dependencies;

in
  assert lockData.lockVersion == 1;
  {}

[nrdxp]

I have a developer at work who was able to help generate a lock file with an sbt plugin, I'll see if I can leverage that to experiment and get something working here 🤞

[zaninime]

@nrdxp how is that different from sbt-dependency-lock?

I like the idea of using lockfiles a lot, but I'm not convinced it should be pushed as being the only option for packaging sbt projects. Not everyone uses sbt-dependency-lock (or equivalent plugins).

[lovesegfault]

@nrdxp how is that different from sbt-dependency-lock?

I like the idea of using lockfiles a lot, but I'm not convinced it should be pushed as being the only option for packaging sbt projects. Not everyone uses sbt-dependency-lock (or equivalent plugins).

I think what's being suggested is that lockfile support be available to those willing/able to use an SBT plugin. I'm certainly not arguing for the ability to just specify the dependencies' combined hash to be removed.

[nrdxp]

Yeah I don't think we'd necessarily lock anybody into it, but using it without a lock is very tedious, since even the slightest change requires regenerating the hash, which for a sizable project can take quite some time on its own. Therefore, it'd be quite nice to have the option, it's good practice anyway, and it might be an encouraging factor to make life easier.

I did a bit of work on this already but I'll be on vacation for most of next week. Hopefully I'll have something solid by the week after unless someone beats me to it 🙂

[jonringer]

not to mention that you almost always have to do a sbt compile to compute a correct FOD, and that can take ~10+ mins on moderate to large projects

[nrdxp]

After a small success with a minimal build, I have run into issues where the information in the lock file isn't enough to reconstruct a url to it from maven. As a specific example,

here is a lock entry with missing scala version information:

{
  "org" : "software.purpledragon",
  "name" : "sbt-dependency-lock",
  "version" : "1.2.0",
  "artifacts" : [
    {
      "name" : "sbt-dependency-lock.jar",
      "hash" : "sha1:f62b30a2bbedd7114910c8bd92f1da1deca13381"
    }
  ],
  "configurations" : [
    "compile",
    "runtime",
    "test"
  ]
}

and here is a url to its artifact:
https://repo1.maven.org/maven2/software/purpledragon/sbt-dependency-lock_2.12_1.0/1.2.0/sbt-dependency-lock-1.2.0.jar

Almost everything thing is there except for the scala version 2.12_1.0. Many of the locked dependencies contain this version as part of the name object above.

[jvdp]

and here is a url to its artifact: repo1.maven.org/maven2/software/purpledragon/sbt-dependency-lock_2.12_1.0/1.2.0/sbt-dependency-lock-1.2.0.jar

Almost everything thing is there except for the scala version 2.12_1.0. Many of the locked dependencies contain this version as part of the name object above.

While you can take Scala and sbt versions from sbt files things become a bit more involved; this URL format actually depends on the repository and not everything is downloaded from Maven.

Would it help to include the list of possible URLs in the lockfile? (Or perhaps separately.)

[zaninime]

The URLs can be easily computed by using the resolvers or fullResolvers key as provided by sbt.

@nrdxp would it be possible for you to check if the ones missing the scala version in the lockfile are all sbt plugins?

[nrdxp]

They are not all plugins no, and some plugins do have this information in their name. For a few more example, kanela-agent doesn't have it, while scalac-scapegoat-plugin does. I am not sure what the connection is. I think @jvdp's suggestion to add a url to the lock file would probably be the easiest solution (at least that I can think of).

[kubukoz]

#5 (comment)

Randomly thought about this and it seems you can actually override the warmup derivation's attrs to exclude actual sources from the build. I imagine we have to run compile only to get the compiler bridge anyway. So we don't need to compile everything! Still, it's painful that every minor change creates a brand new derivation with deps.

Example: https://github.com/kubukoz/nix-milk/blob/7344ba65be8c0e4f6b6c4a697a99e2499dc69ee5/derivation.nix

[nrdxp]

So after a very long and utterly unsuccessful attempt at using sbt's built-in makePom command to generate pom.xml files that can be fed into mvn2nix, I think the best way forward may just be to enhance the sbt-dependency-lock plugin as I mentioned here:
stringbean/sbt-dependency-lock#28 (comment)

The reason why mvn2nix didn't work really boils down to the fact that SBT intetially broke the pom.xml spec for its plugins, so mnv2nix simply can't resolve them. Another issue with that approach is that some plugins are only available via ivy, which mvn2nix knows nothing about.

In order to avoid reimplenting a dependency resolver, this problem is gonna have to be solved inside SBT itself. The changes necessary really aren't that complicated, but due to my lack of familiarity with Scala, and my failed attempts at setting up a usable Scala IDE on NixOS, I haven't been able to give it an honest go just yet. That said, for somebody that knows Scala, they could probably implement the necessary changes to the aforementioned plugin fairly easily.

[nrdxp]

so after recently finding a better dependency fetching command via #8, I decided to give building a lockfile another go. I was able to successfully create a simple JSON lock file for a work project and build the Scala project in Nix without the help of the fixed-output dependency derivation. Unfortunately the project is still a private repo. It will be open sourced eventually, but I thought I'd at least post my lockfile generation script as a gist so you could see how I did it:
https://gist.github.com/nrdxp/33e22cfb7d9afdef5e6c5b909bc6834d

Note: this is run as a makes task

Ideally, I'd like to abstract this logic and add a translator and builder to dream2nix, but I wanted to post this to show that it is possible.

[zaninime]

Closing this issue as the missing lockfile support seem to be related to a deficiency in sbt, rather than in this project.

Also note that implementing lockfile support as part of this project will require its regeneration every time a dependency is added or removed, similarly to updating the hash in the current version. It is not clear if regenerating the lockfile will be a faster process than regenerating the hash, too.

I'm experimenting with some custom sbt plugins, but I couldn't find something that captures everything yet.

[nrdxp]

Agree that this is more of an SBT issue. Just wanted to post this here:
https://github.com/dermetfan/sbt-derivation-lock

I wrote most of the logic for this, but a colleague put it into it's own repo since the one it was living in is still private. I'm already using this for a work project to generate a simple json lock file of all dependencies, though not this version, so I'm not sure if it works as is.

The meat is here:
https://github.com/dermetfan/sbt-derivation-lock/blob/master/lock-deps.nix#L27-L67