Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hakyll dependency on nightly is not cached #3899

Closed
st3ll1s opened this issue Mar 4, 2018 · 13 comments
Closed

hakyll dependency on nightly is not cached #3899

st3ll1s opened this issue Mar 4, 2018 · 13 comments
Labels
Milestone

Comments

@st3ll1s
Copy link

@st3ll1s st3ll1s commented Mar 4, 2018

With the resolver set to resolver: nightly-2018-03-01, a package depending on hakyll always rebuilds haddock-library, pandoc, pandoc-citeproc and hakyll. This does not happen if I set the resolver to resolver: lts-10.7. The rebuilds cause significant slowdowns.

Both pandoc and pandoc-citeproc seem to be extra-deps of hakyll. Maybe this is related?

Steps to reproduce

  1. In stack.yaml, set resolver: nightly-2018-03-01.
  2. Run command stack build.
  3. Run command stack build again.

package.yaml:

name: repro

dependencies:
- base >= 4.7 && < 5
- hakyll

library:
  source-dirs: src

Expected

The second stack build should not build anything again.

Actual

haddock-library, pandoc, pandoc-citeproc and hakyll are rebuilt.

Output of the second stack build --verbose: https://gist.github.com/st3ll1s/3ef40d7d3e5513a31ef031bab129deff

Stack version

root@9aa4322411fe ~# stack --version
Version 1.6.5, Git revision 24ab0d6ff07f28276e082c3ce74dfdeb1a2ca9e9 (5514 commits) x86_64 hpack-0.20.0

Method of installation

curl -sSL https://get.haskellstack.org/ | sh

@st3ll1s
Copy link
Author

@st3ll1s st3ll1s commented Mar 9, 2018

I think this may be caused by haddock-library which is a transitive dependency of hakyll. Just depending on haddock-library suffices to cause rebuilds on every stack build invocation.

@st3ll1s
Copy link
Author

@st3ll1s st3ll1s commented Mar 9, 2018

I found a single commit in Haddock that introduces the change: haskell/haddock@3bf3d4c.
Before to this commit the rebuilds do not happen.
The latest LTS does not include the above commit which explains why this problem did not occur before.

Steps to reproduce

Run stack build and then stack build again.

package.yaml as before.
name: repro

dependencies:
- base >= 4.7 && < 5
- haddock-library

library:
  source-dirs: src

Build is cached

stack.yaml:

resolver: nightly-2018-03-07

extra-deps:
  - git: https://github.com/haskell/haddock.git
    commit: 8064353244842c8a5b04d99cd5e6bb6430a9b51c
    subdirs:
      - haddock-library

Build is not cached

stack.yaml:

resolver: nightly-2018-03-07

extra-deps:
  - git: https://github.com/haskell/haddock.git
    commit: 3bf3d4c95c3b32ff350e127ee087c85d5bbb24c6
    subdirs:
      - haddock-library
@mgsloan
Copy link
Contributor

@mgsloan mgsloan commented Mar 12, 2018

@st3ll1s Thanks for digging into this. Looks like a bug at some level indeed.

My guess is that there is an assumption somewhere that these sub libraries do not alias existing library names like attoparsec. Not sure.

I encourage anyone affected by this to dig into stack's code and see if they can work out exactly what's going on and ideally fix it in the form of a PR.

@mgsloan mgsloan added the type: bug label Mar 12, 2018
@mgsloan mgsloan added this to the P2: Should milestone Mar 12, 2018
@bergus
Copy link

@bergus bergus commented Mar 19, 2018

I experience the same issue with Hakyll. It causes build times of about 15 min, which is unbearable. How can I help to get this fixed? I tried running stack with the verbose flag, but the output didn't indicate anything special.

k-bx referenced this issue in haskell/haddock Mar 19, 2018
A practical benefit is that we can control the build-depends and also
avoid some recompilation between library and test-suite.
@k-bx
Copy link
Contributor

@k-bx k-bx commented Mar 19, 2018

@bergus seems like you were right in pinpointing the commit, it defines library attoparsec which probably somehow causes the rebuilds. I guess there are two options: ask haddock-library maintainers to change this back, or to improve stack to be able to handle this use-case. Not sure if I'll have time to look into this in detail anytime soon, unfortunately.

@st3ll1s
Copy link
Author

@st3ll1s st3ll1s commented Mar 21, 2018

I've tried changing haddock-library's cabal file (st3ll1s/haddock@b3912d7) as @gbaz suggested, but when building haddock-library with stack I still see rebuilds.

stack build --verbose prints Ignoring package z-haddock-library-z-attoparsec-vendored, from (InstalledTo Local, [...]), due to it being unknown to the resolver / extra-deps a few times.

@k-bx
Copy link
Contributor

@k-bx k-bx commented Mar 21, 2018

@st3ll1s can you do something similar to undoing the haskell/haddock@3bf3d4c , e.g. don't introduce any named libraries at all please?

@st3ll1s
Copy link
Author

@st3ll1s st3ll1s commented Mar 21, 2018

@k-bx See st3ll1s/haddock@b29547e. This seems to work, no unnecessary rebuilds anymore!

@gbaz
Copy link

@gbaz gbaz commented Mar 25, 2018

It appears that internal library support was added to stack already (cf #3361 and the resolution in #3430). So I think the problem here is just the rebuild logic, and was likely present from the start. I could get a minimal repro going with the internal-libraries repo given in #3926

Indeed I see a similar message in the verbose logs:

2018-03-24 20:14:30.161561: [debug] Ignoring package z-internal-libraries-z-internal, from (InstalledTo Local,"/..."), due to it being unknown to the resolver / extra-deps.

So it appears that the package is discarded from the check for known/needed packages, and thus forced to always be rebuilt. Since it is rebuilt, this in turn precipitates everything downstream of it being rebuilt.

Since this affects all packages with internal libraries, then it seems like it would be good to fix. Until then, all such packages will have forced rebuilds.

By the way, if someone wants to tackle it, from reviewing 3430, my best guess at where the problem is would be:

(libraryMods,libDotCabalFiles,libWarnings) <- -- FIXME add in sub libraries

The comment seems relevant. (edit: just checked master and confirmed the comment is still there)

@mihaimaruseac
Copy link
Contributor

@mihaimaruseac mihaimaruseac commented Apr 3, 2018

I think I'm going to give it a try and see if I can submit a PR fixing this sometimes this week.

@mihaimaruseac
Copy link
Contributor

@mihaimaruseac mihaimaruseac commented Apr 6, 2018

I've made a candidate PR but it's still work in progress, I'll revisit it after the weekend. At the moment, the internal-libraries above passes, but a hakyll-based project still recompiles haddock-library due to

2018-04-05 21:18:26.547698: [debug] Ignoring package z-haddock-library-z-attoparsec, from (InstalledTo Snap,"/home/mihai/.stack/snapshots/x86_64-linux-tinfo6/lts-11.2/8.2.2/pkgdb/"), due to it being unknown to the resolver / extra-deps.

Most likely, I'll need a similar change to the one for InstalledTo Local, but that seems to conflict with changes for #292 so I have to think more about this.

I'll add more to this after the weekend, don't think I can get some more work done until then.

@mihaimaruseac
Copy link
Contributor

@mihaimaruseac mihaimaruseac commented Apr 6, 2018

Got some time today, I think all is done now and #3955 solves this.

robertjlooby added a commit to robertjlooby/project-euler that referenced this issue Apr 13, 2018
A bug in stack was causing several dependencys to be recompiled on every
build

See: commercialhaskell/stack#3899
@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Apr 17, 2018

Solved by #3955, thanks @mihaimaruseac!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants