add ability to emit a lock file

[johnynek]

the coursier CLI has a resolve option and -t but the output is more useful for humans than tools.

Would you be open to a PR to add the ability to emit a JSON lock file?

I imagine it would list all the constraints that went in (all the artifacts + versions), and then list the minDependencies optionally including sha256 hashes of all the artifacts.

This is suitable for checking into a version control for a fully reproducible description of the classpath used to build.

This protects people from mutable changes in repositories (which are possible) causing hard to diagnose issues (not noticing that the resolution changed).

If this sounds like it would be useful, I can see about sending some PRs to implement this.

[eed3si9n]

I'd be interested as well - sbt/sbt#2989

[johnynek]

Actually a clean implementation of this is probably just a JSON serialization of the Resolution file that is sane enough that humans can read in a code review. If we have Resolution.toJson and Resolution.fromJson then a CLI access of that, I think that's probably what we want, right?

[alexarchambault]

Are you aware of the already existing option to generate a JSON report, for the fetch command:

$ coursier fetch org.typelevel::cats-core:2.1.0 --json-output-file report.json

The resulting file looks like this:

{
  "conflict_resolution": {},
  "dependencies": [
    {
      "coord": "org.scala-lang:scala-library:2.13.1",
      "file": "/home/alexandre/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.1/scala-library-2.13.1.jar",
      "directDependencies": [],
      "dependencies": []
    },
    {
      "coord": "org.typelevel:cats-core_2.13:2.1.0",
      "file": "/home/alexandre/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-core_2.13/2.1.0/cats-core_2.13-2.1.0.jar",
      "directDependencies": [
        "org.scala-lang:scala-library:2.13.1",
        "org.typelevel:cats-kernel_2.13:2.1.0",
        "org.typelevel:cats-macros_2.13:2.1.0"
      ],
      "dependencies": [
        "org.typelevel:cats-macros_2.13:2.1.0",
        "org.typelevel:cats-kernel_2.13:2.1.0",
        "org.scala-lang:scala-library:2.13.1"
      ]
    },
    {
      "coord": "org.typelevel:cats-kernel_2.13:2.1.0",
      "file": "/home/alexandre/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-kernel_2.13/2.1.0/cats-kernel_2.13-2.1.0.jar",                
      "directDependencies": [
        "org.scala-lang:scala-library:2.13.1"
      ],
      "dependencies": [
        "org.scala-lang:scala-library:2.13.1"
      ]
    },
    {
      "coord": "org.typelevel:cats-macros_2.13:2.1.0",
      "file": "/home/alexandre/.cache/coursier/v1/https/repo1.maven.org/maven2/org/typelevel/cats-macros_2.13/2.1.0/cats-macros_2.13-2.1.0.jar",                
      "directDependencies": [
        "org.scala-lang:scala-library:2.13.1"
      ],
      "dependencies": [
        "org.scala-lang:scala-library:2.13.1"
      ]
    }
  ],
  "version": "0.1.0"
}

Could it fit your needs? It was added for pants, and is used by bazel too.

[alexarchambault]

Resolution itself is quite low-level, I'm not sure it's a good idea to serialize it as is. (In some cases, its dependencySet field can have lots of almost identical dependencies, that only differ by their exclusions for example, like when spark dependencies are involved. Some of its fields involve black-box Function1 instances too.)

The current report could be improved though, so that it can be generated from the resolve command, based solely on the results of Resolution.dependencyArtifacts for example, prior to any attempt to actually fetch the artifacts.

[johnynek]

Thanks for the reply.

I was not aware of that option. How is fetch different from resolve? If we fetch a set of arguments will it the report also resolve them down to a single version of each module?

Can fetch be used to generate a lock file as is the goal of the sbt issue and the motivation for my request?

Would you be willing to add http urls and sha256 hash values rather than only having pointers to a file system location?

Next question: the file does not look reproducible (e.g. the order of the artifacts doesn’t seem sorted and is different in the directDependencies vs dependencies). Is that accurate?

Would you be open to a PR to make that file (more) deterministic so it is more suitable for hashing?

[johnynek]

Note I guess this is basically a duplicate of #1223 and we all seem to have the same use case in mind.

[alexarchambault]

How is fetch different from resolve? If we fetch a set of arguments will it the report also resolve them down to a single version of each module?

resolve basically builds a dependency graph, relying only on metadata (POM and ivy.xml). It then prints the full list of dependencies.

fetch does what resolve does (apart from printing dependencies), then actually fetches the artifacts (JARs mostly), and prints the local paths of the artifacts. So it fixes the versions too, like resolve.

Can fetch be used to generate a lock file as is the goal of the sbt issue and the motivation for my request?

I haven't though much about how the JSON report could be used as a lock file from sbt… If its format, modulo some improvements, works fine, I guess it could be used for that. From sbt, we would need some logic to then pick the lock file, and rely on it instead of running an actual resolution.

Would you be willing to add http urls and sha256 hash values rather than only having pointers to a file system location?
Would you be open to a PR to make that file (more) deterministic so it is more suitable for hashing?

Definitely!

[aucampia]

I think what would be quite ideal is if resolve was able to emit json - similar to cs fetch --json-output-file ... - and then for cs fetch to be able to use that, so for example:

cs resolve org.codehaus.groovy:groovy-all:3.0+ --json-output-file /var/tmp/coursier-lock.json
cs fetch --json-input-file /var/tmp/coursier-lock.json

[aucampia]

Actually fetch, launch, bootstrap and install should also be able to operate with a lockfile

[stuhood]

FWIW: Pants uses cs fetch --json-report-file followed by digesting the resulting artifacts to generate a lockfile. At consume time, it uses a cs fetch call and a digest validation step to confirm that the right artifact was fetched.