New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Generate Hackage/LTS nix expressions on the fly rather than have them in the repo #16130
Comments
The main thing I'd want to test is whether your In principle I'm super duper in favor, and this is the thing that I've been wanting to see for over a year. A few thoughts:
cc @edolstra since this is the (kind of) thing I've been bugging him about for a while now, especially around NixOS/nix#52. I'm pretty sure that something resembling this will have to be in a future nixpkgs if we want to keep growing past a certain point. |
Implemented in 2862d27. |
In my own experiments adapting this method to our Qt and KDE packages, I have discovered a possible fatal flaw. Hydra evaluates Nixpkgs in restricted mode, which prevents fetching during evaluation and prevents evaluating Nix expressions from the store. In other words, Hydra can never evaluate a package with an expression generated this way. |
Duh. 😞 |
Oh no! That is a fatal flaw indeed :-( |
So that means Hydra cannot build packages generated with #16005 either. |
I don't think this needs improvements in nix exactly. The question is, do we want hydra to be downloading from arbitrary places on the web or not? If so, we should allow downloads during restricted mode. If not, not. |
@garbas restricted mode also restricts arbitrary filesystem access, which we definitely want on hydra (don't want someone to be able to upload a nixexpr that dumps /etc/passwd on some hydra box for example) |
Alternatively some mechanism to specify a blessed list of downloads, possibly transformed into hydra build inputs. But then that list would need to be protected more than just general nixpkgs access, or else we might as well just enable downloads in restricted mode again. |
The Nix manual states
If I am correct making an exception for just the Nix store would be sufficient. |
No, it's not about accessing store paths in this case, it's about downloads (which aren't documented as part of restricted mode it seems) |
Aren't builds sandboxed from accessing local paths anyway? |
Sure, but without restricted mode evals aren't, and evals copy paths to the store. So I could do |
Sounds like we need a "network-only" mode |
Well, restricted mode exists basically only for hydra, so if we want to allow evals to do arbitrary networking we should just remove networking from being gated by restricted mode. |
I guess it really depends on what the original motivation for restricting networking was. I obviously would prefer it to be allowed to enable exactly this sort of use case, but who knows what I'm missing! |
Import-from-derivation is not forbidden on Hydra. E.g. the Debian and RPM functions use it. However, it's a bad idea because it can cause a significant amount of building at evaluation time. (E.g. if there is a stdenv change and Hydra evaluates a call to Import-from-derivation is however forbidden in read-only mode, so it would break There is an orthogonal issue of whether a call to the builtin |
Here's a transcript from a follow-up convo I had with @edolstra on IRC:
|
Regarding import-from-derivation, we could allow substitution (but not building) in read-only mode. So if |
@edolstra, I like this solution:
The only complication is that we don't have one big fat "auto-haskell.nix" file, but instead we have several hundred small "cabal2nix-foo-x.y.nix" files, where "foo" is some Haskell package and "x.y" is some version number. To get all those files realized, we would have to assign a proper attribute to each of those expressions so that |
I'm closing this issue since we can generate Hackage expressions on the fly, which is the subject of this ticket. Further discussion of the "how to build on Hydra" issue should take place at NixOS/nix#954, IMHO. |
This is the first step towards dropping Stackage support. We keep LTS 6.x around because I don't want to downgrade our default compiler to GHC 7.x, but once LTS 7.x comes out we'll switch our main package set to that and drop Nightly. More details are at: http://permalink.gmane.org/gmane.linux.distributions.nixos/20505 Closes NixOS#14897. Also relevant: - NixOS#16130 - commercialhaskell/stack#2259
@peti brought up an idea earlier on the IRC channel: if
hackage-packages.nix
is automatically generated, why does it need to be in the nix repo? Could we instead generate it as a nix derivation which would contain all versions of packages on hackage.I hacked something together that's very crude, incomplete, and that needs a lot of work before it becomes even remotely serviceable but it illustrates the concept: https://gist.github.com/obadz/347fcf3ef0a86a9dbccbb7b04a80793b
You can inspect the generated nix expressions with:
and you can see that at least one package builds by trying:
Maybe we can move
hackage-package.nix
out ofnixpkgs
. We can probably do something similar for LTS releases (also @peti's idea).Notes:
ab
are generated. That's just to get rapid feedback while iterating.cabal2nix
but I suspect there are more appropriate tools that I'm just not familiar with to do thishaskellPackages
, we'd probably need to have enough packages in "static nix expressions" to buildcabal2nix
or any required tooling written in haskellcc // @acowley @ttuegel @ocharles
The text was updated successfully, but these errors were encountered: