Enable pinning with stack_snapshot

[aherrmann]

depends on #1379

Closes #1310

This implements pinning for stack_snapshot similar to rules_jvm_external's maven_install. By default stack_snapshot works as before, calling stack ls dependencies json to determine package versions and the dependency graph on the fly and calling stack unpack to fetch the package sources. However, if the optional attribute stack_snapshot_json is specified then it is read as a lock file that stores all relevant information to be able to fetch and build the snapshot packages without invoking stack. The lock file can be generated or updated with the script @<workspace>-unpinned//:pin.

This makes re-fetching stack_snapshot repositories faster as Hackage and archive dependencies can be cached in Bazel's repository cache, and stack does not need to update it's database to determine package versions and dependencies.
Closes #1112
Closes #1168

Additionally, this avoids issues due to concurrent invocations of stack which can fail due to a race condition on the Hackage security lock.
Closes #1167

The generated snapshot.json file is meant to be checked into the user's repository. It should not be updated manually, only using bazel run @<workspace>-unpinned//:pin. Similar to rules_jvm_external we validate the contents of the lock file and store and check a checksum to detect a corrupted lock file.

This PR extends the stack_snapshot API docs and adds a comment to examples/WORKSPACE and the WORKSPACE generated by the start script instructing the user how to pin stack_snapshot dependencies. Additionally, this pins the stack_snapshot repositories @stackage, @stackage-zlib, and @ghcide in rules_haskell. The @stackage repository in examples remains unpinned and serves as a test-case for an unpinned stack_snapshot.

Notes

• The resulting stack_snapshot workspaces are identical with the exception of revisions of Cabal files.
• stack ls dependencies json does not expose the Hackage revision of any given package used in the snapshot, only the version. The original Cabal file may not work in the given Stackage snapshot, meaning a later revision is required, but downloading the latest revision is not reproducible. To work around this we pin the current git revision of the all-cabal-hashes repository and download Cabal files from that git revision. This may potentially use a later Hackage revision than used in the Stackage snapshot.
• stack ls dependencies json does not give us access to sha256 of Hackage or archive dependencies. See Feature Request: Expose sha256 in ls dependencies json commercialhaskell/stack#5274 and sha256 and size in ls dependencies json for archive dependencies commercialhaskell/stack#5280. We use the all-cabal-hashes repository to obtain sha256 of Hackage dependencies, and fetch archive dependencies during pinning to determine the hash.
• stack ls dependencies json does not provide subdirectory information of archive dependencies. This change includes a heuristic which expects the Cabal file at the archive root or underneath a top-level directory and defines stripPrefix accordingly.
• Bazel cannot cache git (or hg) dependencies in the repository cache as of now, see Skylark API for accessing repository cache in repository rules bazelbuild/bazel#5086. This implementation keeps using stack unpack for these kinds of dependencies. To avoid issues around stack update this should also be switched to a Bazel implementation in the future.

[Profpatsch]

Hyped for this. It removes the 5–10 second stack overhead that happened every time the bazel daemon was started, right?

[aherrmann]

It removes the 5–10 second stack overhead that happened every time the bazel daemon was started, right?

That's right, to be precise it removes any calls to stack for stack_snapshot that use the lock file and, for now, don't use git or hg dependencies.

[dpulls]

🎉 All dependencies have been resolved !

[hanshoglund]

This looks like a great improvement to me. I have not reviewed the implementation in haskell/cabal.bzl in detail, but the switch to pinned Stackage in the rules_haskell workspace is reassuring enough for a new feature in my opinion.

It would be good to add more regression tests for the unpinned use case, as this is what current projects will see.

Otherwise my concerns are mostly about documentation. I suggest opening an issue to track improvements there.

[Profpatsch]

Looks fairly straight-forward, just a lot of footwork to implement everything. LGTM

[aherrmann]

It would be good to add more regression tests for the unpinned use case, as this is what current projects will see.

Otherwise my concerns are mostly about documentation. I suggest opening an issue to track improvements there.

That's a good idea, I've opened #1385 to that end. Feel free to edit and add to the list.