Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Usage of alternative package indices #4137

Closed
snoyberg opened this issue Jul 7, 2018 · 10 comments
Closed

Usage of alternative package indices #4137

snoyberg opened this issue Jul 7, 2018 · 10 comments
Projects
Milestone

Comments

@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Jul 7, 2018

Stack—like cabal-install—has support for specifying alternative package indices, where the default package index is Hackage, or more specifically, FP Complete's S3-backed mirror. Providing alternative package indices is useful for two completely different use cases in my experience:

  1. Providing for packages that are not on Hackage
  2. Providing for a local mirror of Hackage, such as one behind a corporate firewall

There's no issue at all with the second use case. However, there are some major issues with the first use case:

  • It breaks the meaning of snapshots. foo-1.2.3 means "foo version 1.2.3 as uploaded on Hackage." With an alternative package index, that could mean something quite different.
  • Stack already provides a more direct way of providing non-Hackage packages: via tarball URLs or Git repos in either stack.yaml or custom snapshot files.

I would like to move in the direction of more reliably checking package hashes to ensure that build plan you're getting actually matches what a snapshot references. This may have an impact on how we deal with package indices. So I'm reaching out now to find out: is anyone using alternative package indices in ways they would be worried would be broken by changes here?

@snoyberg snoyberg added this to the P1: Must milestone Jul 7, 2018
@centromere
Copy link

@centromere centromere commented Jul 7, 2018

We use stack in production and would not be impacted by your changes.

@DanBurton
Copy link
Contributor

@DanBurton DanBurton commented Jul 7, 2018

I would like to move in the direction of more reliably checking package hashes to ensure that build plan you're getting actually matches what a snapshot references.

I was under the impression that it already does this. Specifically, I thought that since stackage snapshots specify hashes, that these were used in order to select the desired revision of a package. Is this behavior disabled by the use of alternative package indices?

@DanBurton
Copy link
Contributor

@DanBurton DanBurton commented Jul 7, 2018

I see a 3rd potential use case, which would be something similar to HEAD.hackage, facilitating the development and testing of packages against GHC HEAD (or alpha/rc ghc versions) and corresponding pre-release versions of libraries.

I am unaware of anyone currently using stack's alternative package index in this way, but I'd like for it to be something that people start doing.

@snoyberg
Copy link
Contributor Author

@snoyberg snoyberg commented Jul 7, 2018

That's actually an example of exactly the kind of use case I'd like to avoid: there's no way to ensure reproducibility of builds. I'd rather see a tool that creates custom snapshots that use a specific set of packages from an overlay.

@DanBurton
Copy link
Contributor

@DanBurton DanBurton commented Jul 8, 2018

Good point. I agree 100%

So in such a scenario, the custom snapshot based on an overlay would still need some way to reference things that aren't available on the usual hackage package index. As long as there is some sort of "download by hash", it doesn't really matter where it comes from.

@snoyberg snoyberg added this to Needs review in Pantry Aug 7, 2018
@snoyberg snoyberg closed this in 7aecad8 Aug 26, 2018
Pantry automation moved this from Not merged to Done Aug 26, 2018
vvv added a commit to vvv/stack that referenced this issue Sep 9, 2018
When a link is wrapped and new line starts with '#', Markdown renderers
that line as a header and the result looks ugly.

| [issue
| commercialhaskell#4137](commercialhaskell#4137)
vvv added a commit to vvv/stack that referenced this issue Sep 9, 2018
When a link is wrapped and new line starts with '#', Markdown renders
that line as a header and the result looks ugly.

| [issue
| commercialhaskell#4137](commercialhaskell#4137)
@gridaphobe
Copy link

@gridaphobe gridaphobe commented Mar 27, 2019

Sorry to be so late to the discussion here...

At Bloomberg we have an internal mirror of Hackage that we also publish internal packages to, i.e. we do both (1) and (2). This is very convenient because we have a single place to download packages and browse documentation, we can have internal haddocks cross-link to public packages, etc.

I understand the concern about breaking snapshots due to name clashes, and don't have a problem with checking hashes more aggressively. If, somehow, we got to a point where foo-1.2.3 pointed to a different package in our internal mirror, I would want stack to complain! But we've been careful about package names and haven't had any issues so far, so it would be unfortunate if stack made changes that broke the broader workflow of having an internal Hackage mirror that also contains private packages.

That being said, it's not clear to me whether stack has actually made any changes that would break our workflow. We configure stack with a single package index, our internal mirror, and use the newer 01-index. Package revisions have been an issue for us, the mirror tool doesn't seem to mirror them correctly, but I'd also like to fix that if possible.

@snoyberg
Copy link
Contributor Author

@snoyberg snoyberg commented Mar 28, 2019

As described, I think the changes to Stack should work without problem for the use case you describe.

@ozzzzz
Copy link

@ozzzzz ozzzzz commented Sep 7, 2019

That being said, it's not clear to me whether stack has actually made any changes that would break our workflow. We configure stack with a single package index, our internal mirror, and use the newer 01-index. Package revisions have been an issue for us, the mirror tool doesn't seem to mirror them correctly, but I'd also like to fix that if possible.

Hi, @gridaphobe! Can you write few words more about how you use only one snapshot? Because in BIOCAD we have the same workflow as you, but we still need two package-indices: one public and one private.

@gridaphobe
Copy link

@gridaphobe gridaphobe commented Sep 9, 2019

Hi @ozzzzz, there's not much to it to be honest. We run an internal Hackage server and use the bundled hackage-mirror tool to mirror the public Hackage to our internal server. Then we configure stack to only talk to our internal mirror, which hosts both public and internal packages.

@KaneTW
Copy link

@KaneTW KaneTW commented Dec 28, 2019

Is there a workaround for this when using head.hackage?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Pantry
  
Done
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants