stack_snapshot pinning: Unsupported archive format #1392

teto · 2020-07-12T14:51:35Z

I have this code that I am trying to pin (I successfully pinned another stack_snapshot call):

    stack_snapshot(
        name = "hackage",
        packages = imported_packages,
        local_snapshot = "@buildlib//:hackage/snapshot.yaml",
        flags = {
            "cassava": ["-bytestring--LT-0_10_4"],
        },
        extra_deps = {
            "zlib": ["@zlib.dev//:zlib"],
            "llvm-hs": ["@llvm_9.lib//:lib"],
            "hmatrix": ["@blas//:lib", "@liblapack//:lib"],
            "hmatrix-sundials": ["@sundials//:lib", "@suitesparse//:lib"],
            "nlopt-haskell": ["@nlopt//:lib"],
            "digest": ["@zlib.dev//:zlib"],
        },
        tools = ["@" + name + "//:bin/" + extra_args.get("exe_name", name) for (name, extra_args) in all_haskell_tools.items()],
    )

but it failed with Unsupported archive format:

ERROR: An error occurred during the fetch of repository 'hackage-unpinned':
   Traceback (most recent call last):
	File "/home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl", line 1583
		_pin_packages(repository_ctx, <1 more arguments>)
	File "/home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl", line 1199, in _pin_packages
		fail(<1 more arguments>)
Unsupported archive format at https://github.com/tweag/funflow/archive/1b2c2bdfcc56322e1d31dc0648e08cd776cfc686.tar.gz: Expected funflow.cabal in the root or underneath a top-level directory
ERROR: Traceback (most recent call last):
	File "/home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl", line 1583
		_pin_packages(repository_ctx, <1 more arguments>)
	File "/home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl", line 1199, in _pin_packages
		fail(<1 more arguments>)
Unsupported archive format at https://github.com/tweag/funflow/archive/1b2c2bdfcc56322e1d31dc0648e08cd776cfc686.tar.gz: Expected funflow.cabal in the root or underneath a top-level directory

What can I do to fix/work around it ?

The text was updated successfully, but these errors were encountered:

aherrmann · 2020-07-13T10:21:53Z

stack_snapshot pinning uses stack ls dependencies json under the hood. Unfortunately, stack doesn't expose which subdirectory contains a given package for archive dependencies such as funflow in this example. stack_snapshot works around that limitation by applying a heuristic: It expects the projects Cabal file either in the archive root or a top-level directory. Neither is the case here as the Cabal file is two levels down in funflow-1b2c2bdfcc56322e1d31dc0648e08cd776cfc686/funflow/funflow.cabal.

To work around this issue you can pull funflow in with a separate http_archive rule and point stack_snapshot to it using the vendored_packages attribute. A similar example is provided here with the http_archive defined here.

teto · 2020-07-13T15:39:03Z

thanks for the clear answer. I have tried the following (and several other configurations):

    http_archive(
        name = "funflow",
        build_file_content = """
load("@rules_haskell//haskell:cabal.bzl", "haskell_cabal_library")
load("@hackage//:packages.bzl", "packages")
haskell_cabal_library(
    name = "funflow",
    version = "1.5.0",
    srcs = glob(["**"]),
    deps = packages["funflow"].deps,
)
    """,
        sha256 = "85528bb5b50b4160378f3cbcb73512a039410ce7a0c2b389fb4fc2e94ff6013e",
        strip_prefix = "funflow-1.5.0",
        urls = ["https://github.com/tweag/funflow/archive/v1.5.0.tar.gz"],
    )

    stack_snapshot(
        name = "hackage",
        packages = imported_packages,
        local_snapshot = "@buildlib//:hackage/snapshot.yaml",
        flags = {
            "cassava": ["-bytestring--LT-0_10_4"],
        },
        extra_deps = {
            "zlib": ["@zlib.dev//:zlib"],
            "llvm-hs": ["@llvm_9.lib//:lib"],
            "hmatrix": ["@blas//:lib", "@liblapack//:lib"],
            "hmatrix-sundials": ["@sundials//:lib", "@suitesparse//:lib"],
            "nlopt-haskell": ["@nlopt//:lib"],
            "digest": ["@zlib.dev//:zlib"],
        },
        tools = ["@" + name + "//:bin/" + extra_args.get("exe_name", name) for (name, extra_args) in all_haskell_tools.items()],
        visibility = [ "//visibility:public" ],
		# we vendor funflow since stack_snapshot relies on
		# https://github.com/tweag/rules_haskell/issues/1392
		vendored_packages = {"funflow": "@funflow//:funflow"}
    )

but it fails with

INFO: Call stack for the definition of repository 'hackage' which is a _stack_snapshot (rule definition at /home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl:1836:19):
 - /home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl:2179:5
 - /home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl:2179:5
 - /home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/buildlib/haskell.bzl:121:5
 - /home/teto/nova/jinko/WORKSPACE:55:1
ERROR: An error occurred during the fetch of repository 'hackage':
   Traceback (most recent call last):
	File "/home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl", line 1639
		_resolve_packages(<6 more arguments>)
	File "/home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl", line 1052, in _resolve_packages
		struct(<1 more arguments>)
	File "/home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl", line 1058, in struct
		truly_relativize(<2 more arguments>)
	File "/home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl", line 1059, in truly_relativize
		str(<1 more arguments>)
	File "/home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl", line 1059, in str
		repository_ctx.path(<1 more arguments>)
Not a regular file: /home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/funflow/funflow.cabal

and indeed /home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/funflow/funflow.cabal does not exist.
In the example you've given, I was also surprised that one could do deps = packages["split"].deps, (cyclic dependency) but it seems to work out fine.

aherrmann · 2020-07-13T15:53:19Z

In case of funflow you'll need to configure stripPrefix to strip down to the funflow package directory. I.e.

        strip_prefix = "funflow-1.5.0/funflow",

In the example you've given, I was also surprised that one could do deps = packages["split"].deps, (cyclic dependency) but it seems to work out fine.

Indeed, it's a convenience to avoid having to repeat the list of package dependencies. Bazel is able to separate the downloading of funflow, the execution of stack_snapshot and generation of packages.bzl, and finally the parsing of the BUILD file in @funflow in such a way that this seemingly cyclic dependency doesn't cause an issue.

teto · 2020-07-13T16:14:57Z

Makes sense, I thought about it but had not tried my bad.

I feel bad for following with yet another question sorry but seems like funflow can't see the hackage builds. stack_snapshot doesn't support default_visibility and visibility = [ "//visibility:public" ], doesn't change the outcome thus I am not sure how to work around it.

ERROR: /home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/funflow/BUILD.bazel:4:1: in haskell_cabal_library rule @funflow//:funflow: target '@hackage//:Glob' is not visible from target '@funflow//:funflow'. Check the visibility declaration of the former target if you think the dependency is legitimate
ERROR: /home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/funflow/BUILD.bazel:4:1: in haskell_cabal_library rule @funflow//:funflow: target '@hackage//:async' is not visible from target '@funflow//:funflow'. Check the visibility declaration of the former target if you think the dependency is legitimate
....

Would it make sense for stack_snapshot to adjust visibility for vendored_packages too ?

aherrmann · 2020-07-13T16:28:18Z

Oh, indeed, sorry I forgot to mention that. In this case you'll need to mention the required packages in stack_snapshot to make them publicly visible:

stack_snapshot(
    ...
    packages = [
        ...
        "Glob",
        "async",
    ],
)

Anything that is listed in packages will be made public by stack_snapshot.

teto · 2020-07-13T22:07:00Z

I actually had to remove funflow from packages because of the following error:|

- /home/teto/nova/jinko/WORKSPACE:55:1
ERROR: An error occurred during the fetch of repository 'hackage':
   Traceback (most recent call last):
	File "/home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl", line 1630
		_parse_packages_list(<2 more arguments>)
	File "/home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl", line 1349, in _parse_packages_list
		fail(<1 more arguments>)
Duplicate package 'funflow'. Packages may not be listed in both 'packages' and 'vendored_packages'.
ERROR: Traceback (most recent call last):
	File "/home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl", line 1630
		_parse_packages_list(<2 more arguments>)
	File "/home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl", line 1349, in _parse_packages_list
		fail(<1 more arguments>)
Duplicate package 'funflow'. Packages may not be listed in both 'packages' and 'vendored_packages'.

teto · 2020-09-29T13:01:10Z

I have a similar issue with haskell-language-server master.
Thanks to your instructions in #1433, I had hls (haskell-language-server) 0.4 compiling and running. It had some issues on our codebase though so I wanted to try master and got:

Unsupported archive format at https://github.com/haskell/haskell-language-server/archive/5729c2ea119e9d48c136cc75c377d41fec811dde.tar.gz: Expected hls-plugin-api.cabal in the root or underneath a top-level directory

c99c1888188a3d9285f13d3d2
INFO: SHA256 (https://raw.githubusercontent.com/commercialhaskell/all-cabal-hashes/cda163d5778d02df1160842f383f1927c6db9e44/hie-bios/0.7.1/hie-bios.json) = 0b785a79d25564877b7971d85ca22ee45565d1837a49746b7a46841a902b086b
INFO: SHA256 (https://raw.githubusercontent.com/commercialhaskell/all-cabal-hashes/cda163d5778d02df1160842f383f1927c6db9e44/hie-bios/0.7.1/hie-bios.cabal) = 27b88c6f541e435a41e0c37f94fa286e78eb0065a151a1c599662e6cd0b88c80
INFO: SHA256 (https://github.com/haskell/haskell-language-server/archive/5729c2ea119e9d48c136cc75c377d41fec811dde.tar.gz) = 9ec7fead98d271fd35b994dc7a337713b1d3321b438fdb33f231f2977e0b0f57
INFO: Repository hls-unpinned instantiated at:
  /home/teto/nova/jinko/WORKSPACE:59:20: in <toplevel>
  /home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/buildlib/haskell.bzl:128:34: in jinko_haskell_setup
  /home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/buildlib/hls.bzl:6:19: in setup_haskell_language_server
  /home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl:2231:29: in stack_snapshot
Repository rule _stack_snapshot_unpinned defined at:
  /home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl:1886:43: in <toplevel>
ERROR: An error occurred during the fetch of repository 'hls-unpinned':
   Traceback (most recent call last):
	File "/home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl", line 1647
		_pin_packages(repository_ctx, <1 more arguments>)
	File "/home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl", line 1263, in _pin_packages
		fail(<1 more arguments>)
Unsupported archive format at https://github.com/haskell/haskell-language-server/archive/5729c2ea119e9d48c136cc75c377d41fec811dde.tar.gz: Expected hls-plugin-api.cabal in the root or underneath a top-level directory
ERROR: Traceback (most recent call last):
	File "/home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl", line 1647
		_pin_packages(repository_ctx, <1 more arguments>)
	File "/home/teto/.cache/bazel/_bazel_teto/32070fb26dd3cfe8bd075800f8bb83d1/external/rules_haskell/haskell/cabal.bzl", line 1263, in _pin_packages
		fail(<1 more arguments>)
Unsupported archive format at https://github.com/haskell/haskell-language-server/archive/5729c2ea119e9d48c136cc75c377d41fec811dde.tar.gz: Expected hls-plugin-api.cabal in the root or underneath a top-level directory
INFO: Elapsed time: 37.564s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
FAILED: Build did NOT complete successfully (0 packages loaded)

Indeed hls-plugin-api.cabal is too deep haskell-language-server-5729c2ea119e9d48c136cc75c377d41fec811dde/hls-plugin-api/hls-plugin-api.cabal but can't we just strip the haskell-language-server-5729c2ea119e9d48c136cc75c377d41fec811dde ? it should be doable to extract in a current folder or just move into that folder to not consider it as top-level ?

aherrmann · 2020-11-09T17:43:35Z

@teto I gave fixing this a shot ini #1440

We want to work around an issue with packages not at the top-level. It looks as though there's some issue in the newer versions of `rules_haskell` for Haskell packages that aren't at the top-level. This comment gives an overview of the problem: tweag/rules_haskell#1392 (comment). To paraphrase, Haskell packages in a sub-directory don't really work with `stack ls dependencies json` at the moment. Since we want to update `rules_haskell` to a newer version, we need to address this. We can make this change now without updating the version so that the version bump of `rules_haskell` has a smaller diff. In all likelihood, this issue will be fixed at some point. Haskell packages in a sub-directory are fairly common. Until that day, we follow most of the workaround laid out in that GitHub issue: pull in the package as a `bazel` `http_archive` rule, point `stack_snapshot` to the package as a `vendored_package`. The part we do a bit differently is not stripping down to the sub-directory. Instead, we treat the whole archive as one thing and locate the files based on where they live from the top-level. We take this approach primarily because we're going to pull in both `purescript-ast` and `purescript-cst`. Since the commit we're using doesn't have a `lib/purescript-cst/purescript-cst.cabal` file, we have to patch it to add that.

We want to work around an issue with packages not at the top-level. This follows the change we made for `purescript-cst` in 8f77f15f5d388b6a17c6a216ea1644caeda609af. I.e. addressing the problem of Haskell packages in a sub-directory not really working with `stack ls dependencies json` at the moment: tweag/rules_haskell#1392 (comment).

We want to work around an issue with packages not at the top-level. It looks as though there's some issue in the newer versions of `rules_haskell` for Haskell packages that aren't at the top-level. This comment gives an overview of the problem: tweag/rules_haskell#1392 (comment). To paraphrase, Haskell packages in a sub-directory don't really work with `stack ls dependencies json` at the moment. Since we want to update `rules_haskell` to a newer version, we need to address this. We can make this change now without updating the version so that the version bump of `rules_haskell` has a smaller diff. In all likelihood, this issue will be fixed at some point. Haskell packages in a sub-directory are fairly common. Until that day, we follow most of the workaround laid out in that GitHub issue: pull in the package as a `bazel` `http_archive` rule, point `stack_snapshot` to the package as a `vendored_package`. The part we do a bit differently is not stripping down to the sub-directory. Instead, we treat the whole archive as one thing and locate the files based on where they live from the top-level. We take this approach primarily because we're going to pull in both `purescript-ast` and `purescript-cst`. Since the commit we're using doesn't have a `lib/purescript-cst/purescript-cst.cabal` file, we have to patch it to add that.

We want to work around an issue with packages not at the top-level. This follows the change we made for `purescript-cst` in 8f77f15f5d388b6a17c6a216ea1644caeda609af. I.e. addressing the problem of Haskell packages in a sub-directory not really working with `stack ls dependencies json` at the moment: tweag/rules_haskell#1392 (comment).

We want to work around an issue with packages not at the top-level. It looks as though there's some issue in the newer versions of `rules_haskell` for Haskell packages that aren't at the top-level. This comment gives an overview of the problem: tweag/rules_haskell#1392 (comment). To paraphrase, Haskell packages in a sub-directory don't really work with `stack ls dependencies json` at the moment. Since we want to update `rules_haskell` to a newer version, we need to address this. We can make this change now without updating the version so that the version bump of `rules_haskell` has a smaller diff. In all likelihood, this issue will be fixed at some point. Haskell packages in a sub-directory are fairly common. Until that day, we follow most of the workaround laid out in that GitHub issue: pull in the package as a `bazel` `http_archive` rule, point `stack_snapshot` to the package as a `vendored_package`. The part we do a bit differently is not stripping down to the sub-directory. Instead, we treat the whole archive as one thing and locate the files based on where they live from the top-level. We take this approach primarily because we're going to pull in both `purescript-ast` and `purescript-cst`. Since the commit we're using doesn't have a `lib/purescript-cst/purescript-cst.cabal` file, we have to patch it to add that.

We want to work around an issue with packages not at the top-level. This follows the change we made for `purescript-cst` in 8f77f15f5d388b6a17c6a216ea1644caeda609af. I.e. addressing the problem of Haskell packages in a sub-directory not really working with `stack ls dependencies json` at the moment: tweag/rules_haskell#1392 (comment).

teto added the type: bug label Jul 12, 2020

This was referenced Jul 14, 2020

Feature Request: Expose subdirectory of archive dependencies in ls dependencies json commercialhaskell/stack#5345

Open

stack_snapshot pinning improvements #1385

Open

aherrmann mentioned this issue Nov 9, 2020

Fix stack_snapshot pinning unsupported archive format #1440

Merged

mergify bot closed this as completed in #1440 Dec 1, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stack_snapshot pinning: Unsupported archive format #1392

stack_snapshot pinning: Unsupported archive format #1392

teto commented Jul 12, 2020

aherrmann commented Jul 13, 2020

teto commented Jul 13, 2020

aherrmann commented Jul 13, 2020

teto commented Jul 13, 2020

aherrmann commented Jul 13, 2020

teto commented Jul 13, 2020

teto commented Sep 29, 2020

aherrmann commented Nov 9, 2020

stack_snapshot pinning: Unsupported archive format #1392

stack_snapshot pinning: Unsupported archive format #1392

Comments

teto commented Jul 12, 2020

aherrmann commented Jul 13, 2020

teto commented Jul 13, 2020

aherrmann commented Jul 13, 2020

teto commented Jul 13, 2020

aherrmann commented Jul 13, 2020

teto commented Jul 13, 2020

teto commented Sep 29, 2020

aherrmann commented Nov 9, 2020