Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move configure information for local packages back to .stack-work to improve caching #4893

Closed
jdreaver opened this issue Jun 20, 2019 · 4 comments

Comments

Projects
None yet
4 participants
@jdreaver
Copy link

commented Jun 20, 2019

Hello!

In stack 2.X, it looks like local configuration information for packages was moved to ~/.stack/stack.sqlite3. This breaks a common pattern for caching Haskell dependencies separate from local builds in CI. Our CI caches used to work like this:

  1. Build all Haskell dependencies
  2. Cache ~/.stack
  3. Build our local Haskell packages
  4. Cache all .stack-work directories

Separate caches are really nice because we don't have to pay to re-cache ~/.stack if deps didn't change, and it is easy to share the dependency caches across branches.

Now that local configuration information for packages is in ~/.stack, we had to combine our dependency and local caches, which means we have to cache/uncache ~/.stack (or some yet-undetermined subset of ~/.stack, maybe just the SQLite file?) after building local packages. This means an extra couple minutes wasted in CI, even if dependencies didn't change.

It would be great to keep everything related to local packages outside of ~/.stack so we can regain this nice caching setup, and also maintain the conceptual simplicity of "global/dependency stuff lives in ~/.stack, while everything local to a project lives in .stack-work".

After talking to @borsboom and @snoyberg, it appears this was done out of simplicity so stack only needed to handle one SQLite database, which certainly makes sense. However, they indicated that having local configuration stay in .stack-work and out of ~/.stack would be a good reason to add a bit of complexity and have multiple databases, hence me opening this issue.


For some additional context, here is what happens when building a local package if ~/.stack/stack.sqlite3 is nuked, which replicates what happens if you don't cache ~/.stack in CI:

backend $ stack build core --fast            
backend $ rm ~/.stack/stack.sqlite3*
backend $ stack build core --fast
core-0.0.0: unregistering (old configure information not found)
sendgrid-client-0.0.0: unregistering (Dependency being unregistered: core-0.0.0)
core> configure (lib)

Stack rebuilds local packages from scratch when this happens.

@brandon-leapyear

This comment has been minimized.

Copy link
Contributor

commented Jun 21, 2019

Here's another similar CI workflow that we do:

  1. CI build step:
    • install zlib-devel
    • restore cached ~/.stack
    • stack build --only-dependencies --test
    • save ~/.stack if nothing was restored in the first step
    • stack build
    • save ~/.stack-work
  2. CI test step: (different container)
    • restore cached ~/.stack
    • restore saved ~/.stack-work
    • stack test
my-project: : unregistering (old configure information not found)
> /usr/bin/ld.gold: error: cannot find -lz
> collect2: error: ld returned 1 exit status
> `gcc' failed in phase `Linker'. (Exit code: 1)

As a developer, I would expect stack test to "just" find the test executable and run it. Yes, it's an alias for build --test and so have the same semantics as build with regard to configuring. But behaviorly, it's confusing, especially for people new to stack

snoyberg added a commit that referenced this issue Jun 23, 2019

@snoyberg

This comment has been minimized.

Copy link
Contributor

commented Jun 23, 2019

See #4898

snoyberg added a commit that referenced this issue Jun 26, 2019

snoyberg added a commit that referenced this issue Jun 26, 2019

Merge pull request #4898 from commercialhaskell/4893-local-database
Split up user and project storage (fixes #4893)
@hdgarrood

This comment has been minimized.

Copy link

commented Jul 10, 2019

Should this be closed now that #4898 has been merged? Using a stack version from stack upgrade --git I observed the desired behaviour i.e. I was able to cache .stack-work directories as described in the original issue description.

@snoyberg

This comment has been minimized.

Copy link
Contributor

commented Jul 11, 2019

This will close automatically when we merge stable into master, but we may as well close it now, thanks!

@snoyberg snoyberg closed this Jul 11, 2019

Javran added a commit to Javran/ProjectEuler that referenced this issue Jul 20, 2019

remove before_cache section.
Now that caching is fixed, this is no longer necessary.

context: commercialhaskell/stack#4893
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.