stack solver is broken

[sjakobi]

General summary/comments (optional)

stack solver --update-config doesn't seem to work with either cabal-install-1.22.9.0 or cabal-install-1.24.0.1. My hypothesis is that the problem is caused by cabal not seeing the package index.

stack v1.2.1 @ 0a96df works without issue.

Steps to reproduce

For example:

$ stack --resolver lts-7.14 new solver-repro
$ cd solver-repro
# Add "Cartesian" to the library build-depends in the cabal file
$ stack solver --update-config
Using configuration file: stack.yaml
Using cabal packages:
- solver-repro.cabal

Using resolver: lts-7.14
Using compiler: ghc-8.0.1
Asking cabal to calculate a build plan...
Trying with packages from lts-7.14 as hard constraints...
Attempt failed.

>>>> Cabal errors begin
Warning: The package list for 'Hackage' does not exist. Run 'cabal update' to
download it.
Warning: Constraint refers to an unknown package: AC-Vector ==2.3.2 (main
config /tmp/cabal-solver13270/cabal.config)
cabal: Could not resolve dependencies:
trying: solver-repro-0.1.0.0 (user goal)
next goal: Cartesian (dependency of solver-repro-0.1.0.0)
Dependency tree exhaustively searched.
<<<< Cabal errors end

Retrying with packages from lts-7.14 as preferences...
Attempt failed.

>>>> Cabal errors begin
Warning: The package list for 'Hackage' does not exist. Run 'cabal update' to
download it.
Warning: Preference refers to an unknown package: AC-Vector ==2.3.2
cabal: Could not resolve dependencies:
trying: solver-repro-0.1.0.0 (user goal)
next goal: Cartesian (dependency of solver-repro-0.1.0.0)
Dependency tree exhaustively searched.
<<<< Cabal errors end

*** Failed to arrive at a workable build plan.

Solver could not resolve package dependencies.
You can try the following:
    - Update external packages with 'stack update' and try again.
    - Tweak stack.yaml and try again
        - Remove any unnecessary packages.
        - Add any missing remote packages.
        - Add extra dependencies to guide solver.
        - Adjust resolver.

Expected

stack solver --update-config adds a version of Cartesian to the extra-deps and the following stack build succeeds.

Actual

stack solver --update-config fails. cabal's suggestion to cabal update doesn't help either, nor does stack update.

Here is the output of stack solver in verbose mode.

Stack version

$ cabal --version
cabal-install version 1.24.0.1
compiled using version 1.24.1.0 of the Cabal library 
$ stack --version
Version 1.3.3, Git revision e5987cdd297516e0a111243b45302dc0326e4440 (dirty) (4447 commits) x86_64 hpack-0.15.0

Method of installation

Built from source with the official stack.yaml.

[sjakobi]

Can't repro the issue with stack-1.3.2, so the bug seems to have been introduced recently.

[mgsloan]

I've investigated a bit, and this appears to have been introduced when switching to the new 01-index.tar.gz (new hackage-security format) - efdfafa#diff-b8b3eca5371c1446794562093981903cL252

Not sure why it doesn't work with cabal-install-1.24, but I would indeed expect this to break with earlier cabal versions. Pinging @snoyberg as he implemented the change, and likely has more insight.

[snoyberg]

Good catch, thanks. What do you think if, after downloading 01-index.tar.gz, we simply copy it over to 00-index.tar.gz?

[mgsloan]

Hmm, aren't the formats different? Possibly worth a shot!

[snoyberg]

They're different in that 01-index.tar.gz has a few extra files and will include the historic .cabal files before revisions, but I don't think that will cause any problems. We could reconstruct the 00-index.tar.gz file, but AFAICT it's not necessary.

I also asked about this on Twitter: https://twitter.com/snoyberg/status/817236138742185984. But you're absolutely right about older cabals still needing the old file.

[mgsloan]

Looks like it will indeed happily use the 01-index.tar.gz renamed to 00-index.tar.gz! I've opened a PR for it #2901

[ezyang]

Support for 01-index was added in dc889b17fef7cdd8eac5304d44863e67408e0ad4 and ported to 1.24 branch in 9abda45688947c86137dc0b26ee3992c6412ae2c, so it should be out with 1.24.0.1 (CC @hvr who contributed this patch). I don't really understand the moving parts here but see haskell/cabal#3854 for some reasons why you should avoid moving 01-index.tar.gz on top of 00-index.tar.gz, if people are also using a pre-security version of cabal-install (but maybe you are placing the index in a non-standard location so it's not an issue.)

[snoyberg]

Thanks Edward. I _think_ that specific issue shouldn't be relevant (since we're never asking cabal to do an update, and we're doing everything in a temporary location).

…

On Fri, Jan 6, 2017 at 9:25 AM, Edward Z. Yang ***@***.***> wrote: Support for 01-index was added in dc889b17fef7cdd8eac5304d44863e67408e0ad4 and ported to 1.24 branch in 9abda45688947c86137dc0b26ee3992c6412ae2c, so it should be out with 1.24.0.1 (CC @hvr <https://github.com/hvr> who contributed this patch). I don't really understand the moving parts here but see haskell/cabal#3854 <haskell/cabal#3854> for some reasons why you should avoid moving 01-index.tar.gz on top of 00-index.tar.gz, if people are also using a pre-security version of cabal-install (but maybe you are placing the index in a non-standard location so it's not an issue.) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2888 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADBB19_jQ10qLHY6qX92o9dkxUPy9sCks5rPexxgaJpZM4LZJDE> .

[hvr]

Re cabal-1.24 it's quite simple: everything's in place in cabal 1.24.0.2 for index-01.tar.gz (i.e. crypto keys etc), but the default config for cabal-1.24 is

repository hackage.haskell.org
  url: http://hackage.haskell.org/
  secure: False

while for cabal-1.25+ it is

repository hackage.haskell.org
  url: http://hackage.haskell.org/
  secure: True

IOW, we didn't enable it by default yet because we were being cautious and wanted to deploy this in a more controlled way (all that's needed is toggling a boolean flag in the config) and have a full release cycle time to get feedback on issues before hitting everyone with it. On the other hand, in retrospect this may have been too slow.

[mgsloan]

Closing this issue, since it was resolved in #2901