New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--only-dependencies instead of --only-snapshot #387

Closed
gregwebs opened this Issue Jun 23, 2015 · 12 comments

Comments

Projects
None yet
3 participants
@gregwebs
Contributor

gregwebs commented Jun 23, 2015

As a stack user, I don't know what the terminology "snapshot" means. Also, I am not sure if it means only the resolver or if extra-deps are included. The behavior that I would like is for extra-deps to be included (well at least the ones that are from hackage tip, maybe not a local directory) so it is more similar to cabal install --only-dependencies.

@gregwebs gregwebs changed the title from better name for --only-snapshot to --only-dependencies instead of --only-snapshot Jun 23, 2015

@gregwebs

This comment has been minimized.

Contributor

gregwebs commented Jun 23, 2015

--only-snapshot ignores extra-deps that are just packages from hackage instead of the resolver. For my continuous integration build I need a --only-dependencies flag that install all dependencies. So I am changing the ticket name.

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Jun 24, 2015

See the discussion in #310 to see how we got to where we are now.

@gregwebs

This comment has been minimized.

Contributor

gregwebs commented Jun 24, 2015

#310 seems to say that --only-dep is used for ci builds. This ticket is to take the concept to its logical conclusion of installing extra deps

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Jun 24, 2015

Sorry, I just realized that some of that discussion happened outside of the issue. The basic idea is: there are weird corner cases that come into play if we try to install only dependencies, specifically because of cycles between dependencies and local packages. Consider working on the wai package: some of the dependencies (like wai-logger) depend on non-dependencies (wai). Do we install wai and wai-logger, or not? The obvious cut-off was to just install whatever would go into the snapshot database and drop anything local/custom.

That's not to say that there isn't value in doing something better, I'm just explaining how we got to the current place. If someone wants to put in the time to figure out a better solution, that would be good. But it doesn't seem like this is high-value enough to be worth it (unless you have use cases I'm not anticipating).

@snoyberg snoyberg added this to the Later improvements milestone Jun 24, 2015

@gregwebs

This comment has been minimized.

Contributor

gregwebs commented Jun 24, 2015

For wai-logger, sure you cannot install that one ahead of time. But that seems like a rare corner case (that should be handled properly). The usual case is just wanting packages from the head of hackage instead of the LTS. And for some projects, that could mean a lot or packages.

For us it is a big deal because even though we only have a few packages, it slows down every CI build. We need a step to cache as much as possible, and obviously we don't want to cache the actual build.

However, I think I can work around this for now by just manually installing the extra-deps.

@gregwebs

This comment has been minimized.

Contributor

gregwebs commented Jun 24, 2015

Here is my current workaround to install everything in extra-deps:

# install extra-deps from stack.yaml
# This is hacky, there is probably a better way
# look for the leading '-' in the yaml file and the ending '-' with a version number
stack install $(cat stack.yaml | grep -e '^\s*-.*-[0-9]' | awk '{print $2}' | xargs)
@zimbatm

This comment has been minimized.

zimbatm commented Jul 6, 2015

In my scenario the project source files aren't available during the build step. Basically I want to build all the dependencies except the top-level ones.

stack build --only-snapshot
# Install extra-deps
grep -e '^\s*-.*-[0-9]' stack.yaml | awk '{print $2}' | xargs stack build
# Install github dependencies
grep 'git: git@github.com:' stack.yaml | sed -e 's/.*:.*\/\(.*\).git.*/\1/g' | xargs stack build

Instead of --only-snapshot having a --excluded <list-of-packages> would also work for me.

@gregwebs

This comment has been minimized.

Contributor

gregwebs commented Jul 17, 2015

Another related issue for me is that the extra dependencies are installed into the .stack-work directory: this makes them difficult to cache between CI builds. Perhaps more "nix-like" package management can solve this issue, although I could also solve it with a custom snapshot.

@gregwebs

This comment has been minimized.

Contributor

gregwebs commented Jul 25, 2015

We are solving this issue by creating a custom snapshot. That is a good enough solution for me. Perhaps this workflow should just be documented when the next release is made.

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Aug 9, 2015

See plan laid out at: #651 (comment)

@snoyberg snoyberg closed this Aug 9, 2015

snoyberg added a commit that referenced this issue Aug 11, 2015

@snoyberg

This comment has been minimized.

Contributor

snoyberg commented Aug 11, 2015

Implemented for #651, please review PR #753.

@gregwebs

This comment has been minimized.

Contributor

gregwebs commented Aug 12, 2015

This is working great for us! We are also going to create a custom snapshot now and watch our build time plummet :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment