New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hakyll dependency on nightly is not cached #3899

Closed
st3ll1s opened this Issue Mar 4, 2018 · 13 comments

Comments

Projects
None yet
7 participants
@st3ll1s

st3ll1s commented Mar 4, 2018

With the resolver set to resolver: nightly-2018-03-01, a package depending on hakyll always rebuilds haddock-library, pandoc, pandoc-citeproc and hakyll. This does not happen if I set the resolver to resolver: lts-10.7. The rebuilds cause significant slowdowns.

Both pandoc and pandoc-citeproc seem to be extra-deps of hakyll. Maybe this is related?

Steps to reproduce

  1. In stack.yaml, set resolver: nightly-2018-03-01.
  2. Run command stack build.
  3. Run command stack build again.

package.yaml:

name: repro

dependencies:
- base >= 4.7 && < 5
- hakyll

library:
  source-dirs: src

Expected

The second stack build should not build anything again.

Actual

haddock-library, pandoc, pandoc-citeproc and hakyll are rebuilt.

Output of the second stack build --verbose: https://gist.github.com/st3ll1s/3ef40d7d3e5513a31ef031bab129deff

Stack version

root@9aa4322411fe ~# stack --version
Version 1.6.5, Git revision 24ab0d6ff07f28276e082c3ce74dfdeb1a2ca9e9 (5514 commits) x86_64 hpack-0.20.0

Method of installation

curl -sSL https://get.haskellstack.org/ | sh

@st3ll1s

This comment has been minimized.

st3ll1s commented Mar 9, 2018

I think this may be caused by haddock-library which is a transitive dependency of hakyll. Just depending on haddock-library suffices to cause rebuilds on every stack build invocation.

@st3ll1s

This comment has been minimized.

st3ll1s commented Mar 9, 2018

I found a single commit in Haddock that introduces the change: haskell/haddock@3bf3d4c.
Before to this commit the rebuilds do not happen.
The latest LTS does not include the above commit which explains why this problem did not occur before.

Steps to reproduce

Run stack build and then stack build again.

package.yaml as before.
name: repro

dependencies:
- base >= 4.7 && < 5
- haddock-library

library:
  source-dirs: src

Build is cached

stack.yaml:

resolver: nightly-2018-03-07

extra-deps:
  - git: https://github.com/haskell/haddock.git
    commit: 8064353244842c8a5b04d99cd5e6bb6430a9b51c
    subdirs:
      - haddock-library

Build is not cached

stack.yaml:

resolver: nightly-2018-03-07

extra-deps:
  - git: https://github.com/haskell/haddock.git
    commit: 3bf3d4c95c3b32ff350e127ee087c85d5bbb24c6
    subdirs:
      - haddock-library
@mgsloan

This comment has been minimized.

Collaborator

mgsloan commented Mar 12, 2018

@st3ll1s Thanks for digging into this. Looks like a bug at some level indeed.

My guess is that there is an assumption somewhere that these sub libraries do not alias existing library names like attoparsec. Not sure.

I encourage anyone affected by this to dig into stack's code and see if they can work out exactly what's going on and ideally fix it in the form of a PR.

@mgsloan mgsloan added the type: bug label Mar 12, 2018

@mgsloan mgsloan added this to the P2: Should milestone Mar 12, 2018

@bergus

This comment has been minimized.

bergus commented Mar 19, 2018

I experience the same issue with Hakyll. It causes build times of about 15 min, which is unbearable. How can I help to get this fixed? I tried running stack with the verbose flag, but the output didn't indicate anything special.

k-bx referenced this issue in haskell/haddock Mar 19, 2018

Refactor .cabal to use sub-lib for vendored lib
A practical benefit is that we can control the build-depends and also
avoid some recompilation between library and test-suite.
@k-bx

This comment has been minimized.

Contributor

k-bx commented Mar 19, 2018

@bergus seems like you were right in pinpointing the commit, it defines library attoparsec which probably somehow causes the rebuilds. I guess there are two options: ask haddock-library maintainers to change this back, or to improve stack to be able to handle this use-case. Not sure if I'll have time to look into this in detail anytime soon, unfortunately.

@st3ll1s

This comment has been minimized.

st3ll1s commented Mar 21, 2018

I've tried changing haddock-library's cabal file (st3ll1s/haddock@b3912d7) as @gbaz suggested, but when building haddock-library with stack I still see rebuilds.

stack build --verbose prints Ignoring package z-haddock-library-z-attoparsec-vendored, from (InstalledTo Local, [...]), due to it being unknown to the resolver / extra-deps a few times.

@k-bx

This comment has been minimized.

Contributor

k-bx commented Mar 21, 2018

@st3ll1s can you do something similar to undoing the haskell/haddock@3bf3d4c , e.g. don't introduce any named libraries at all please?

@st3ll1s

This comment has been minimized.

st3ll1s commented Mar 21, 2018

@k-bx See st3ll1s/haddock@b29547e. This seems to work, no unnecessary rebuilds anymore!

@gbaz

This comment has been minimized.

gbaz commented Mar 25, 2018

It appears that internal library support was added to stack already (cf #3361 and the resolution in #3430). So I think the problem here is just the rebuild logic, and was likely present from the start. I could get a minimal repro going with the internal-libraries repo given in #3926

Indeed I see a similar message in the verbose logs:

2018-03-24 20:14:30.161561: [debug] Ignoring package z-internal-libraries-z-internal, from (InstalledTo Local,"/..."), due to it being unknown to the resolver / extra-deps.

So it appears that the package is discarded from the check for known/needed packages, and thus forced to always be rebuilt. Since it is rebuilt, this in turn precipitates everything downstream of it being rebuilt.

Since this affects all packages with internal libraries, then it seems like it would be good to fix. Until then, all such packages will have forced rebuilds.

By the way, if someone wants to tackle it, from reviewing 3430, my best guess at where the problem is would be:

(libraryMods,libDotCabalFiles,libWarnings) <- -- FIXME add in sub libraries

The comment seems relevant. (edit: just checked master and confirmed the comment is still there)

@mihaimaruseac

This comment has been minimized.

Contributor

mihaimaruseac commented Apr 3, 2018

I think I'm going to give it a try and see if I can submit a PR fixing this sometimes this week.

@mihaimaruseac

This comment has been minimized.

Contributor

mihaimaruseac commented Apr 6, 2018

I've made a candidate PR but it's still work in progress, I'll revisit it after the weekend. At the moment, the internal-libraries above passes, but a hakyll-based project still recompiles haddock-library due to

2018-04-05 21:18:26.547698: [debug] Ignoring package z-haddock-library-z-attoparsec, from (InstalledTo Snap,"/home/mihai/.stack/snapshots/x86_64-linux-tinfo6/lts-11.2/8.2.2/pkgdb/"), due to it being unknown to the resolver / extra-deps.

Most likely, I'll need a similar change to the one for InstalledTo Local, but that seems to conflict with changes for #292 so I have to think more about this.

I'll add more to this after the weekend, don't think I can get some more work done until then.

@mihaimaruseac

This comment has been minimized.

Contributor

mihaimaruseac commented Apr 6, 2018

Got some time today, I think all is done now and #3955 solves this.

robertjlooby added a commit to robertjlooby/project-euler that referenced this issue Apr 13, 2018

Bump down to stack lts-10.7
A bug in stack was causing several dependencys to be recompiled on every
build

See: commercialhaskell/stack#3899
@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Apr 17, 2018

Solved by #3955, thanks @mihaimaruseac!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment