Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move configure information for local packages back to .stack-work to improve caching #4893

jdreaver opened this issue Jun 20, 2019 · 4 comments


Copy link

@jdreaver jdreaver commented Jun 20, 2019


In stack 2.X, it looks like local configuration information for packages was moved to ~/.stack/stack.sqlite3. This breaks a common pattern for caching Haskell dependencies separate from local builds in CI. Our CI caches used to work like this:

  1. Build all Haskell dependencies
  2. Cache ~/.stack
  3. Build our local Haskell packages
  4. Cache all .stack-work directories

Separate caches are really nice because we don't have to pay to re-cache ~/.stack if deps didn't change, and it is easy to share the dependency caches across branches.

Now that local configuration information for packages is in ~/.stack, we had to combine our dependency and local caches, which means we have to cache/uncache ~/.stack (or some yet-undetermined subset of ~/.stack, maybe just the SQLite file?) after building local packages. This means an extra couple minutes wasted in CI, even if dependencies didn't change.

It would be great to keep everything related to local packages outside of ~/.stack so we can regain this nice caching setup, and also maintain the conceptual simplicity of "global/dependency stuff lives in ~/.stack, while everything local to a project lives in .stack-work".

After talking to @borsboom and @snoyberg, it appears this was done out of simplicity so stack only needed to handle one SQLite database, which certainly makes sense. However, they indicated that having local configuration stay in .stack-work and out of ~/.stack would be a good reason to add a bit of complexity and have multiple databases, hence me opening this issue.

For some additional context, here is what happens when building a local package if ~/.stack/stack.sqlite3 is nuked, which replicates what happens if you don't cache ~/.stack in CI:

backend $ stack build core --fast            
backend $ rm ~/.stack/stack.sqlite3*
backend $ stack build core --fast
core-0.0.0: unregistering (old configure information not found)
sendgrid-client-0.0.0: unregistering (Dependency being unregistered: core-0.0.0)
core> configure (lib)

Stack rebuilds local packages from scratch when this happens.

Copy link

@brandon-leapyear brandon-leapyear commented Jun 21, 2019

Here's another similar CI workflow that we do:

  1. CI build step:
    • install zlib-devel
    • restore cached ~/.stack
    • stack build --only-dependencies --test
    • save ~/.stack if nothing was restored in the first step
    • stack build
    • save ~/.stack-work
  2. CI test step: (different container)
    • restore cached ~/.stack
    • restore saved ~/.stack-work
    • stack test
my-project: : unregistering (old configure information not found)
> /usr/bin/ error: cannot find -lz
> collect2: error: ld returned 1 exit status
> `gcc' failed in phase `Linker'. (Exit code: 1)

As a developer, I would expect stack test to "just" find the test executable and run it. Yes, it's an alias for build --test and so have the same semantics as build with regard to configuring. But behaviorly, it's confusing, especially for people new to stack

snoyberg added a commit that referenced this issue Jun 23, 2019
Copy link

@snoyberg snoyberg commented Jun 23, 2019

See #4898

snoyberg added a commit that referenced this issue Jun 26, 2019
snoyberg added a commit that referenced this issue Jun 26, 2019
Split up user and project storage (fixes #4893)
Copy link

@hdgarrood hdgarrood commented Jul 10, 2019

Should this be closed now that #4898 has been merged? Using a stack version from stack upgrade --git I observed the desired behaviour i.e. I was able to cache .stack-work directories as described in the original issue description.

Copy link

@snoyberg snoyberg commented Jul 11, 2019

This will close automatically when we merge stable into master, but we may as well close it now, thanks!

@snoyberg snoyberg closed this Jul 11, 2019
Javran pushed a commit to Javran/ProjectEuler that referenced this issue Jul 20, 2019
Now that caching is fixed, this is no longer necessary.

context: commercialhaskell/stack#4893
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants