Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC 0040] "Ret-cont" recursive Nix #40

Closed
wants to merge 21 commits into from
Closed

Conversation

@Ericson2314
Copy link
Member

@Ericson2314 Ericson2314 commented Feb 6, 2019

"Ret-cont" recursive Nix is a restricted form of recursive Nix, where one builds a derivations instead of executing builds during the builds. This avoids some platform-specific contortions relating to nested sandboxing. More importantly, it prevents imperative and overly linear direct-style build scripts; easy to write but throwing away the benefits of Nix.

Rendered

@Ericson2314 Ericson2314 changed the title "Ret-cont" recursive Nix [RFC 0040]: "Ret-cont" recursive Nix Feb 6, 2019
@Profpatsch
Copy link
Member

@Profpatsch Profpatsch commented Feb 6, 2019

I don’t entirely understand the proposal.

From experience, naïvely recursive systems are usually extremely constrained, so I can see that problem. You should split out the problem with the current proposal from “Motivation” into its own section and start with that.

Can you frame the idea in terms of already existing systems? Scheme’s call/cc comes to mind, but from the description it is unclear to me whether the ideas are similar. We shouldn’t implement systems like that without looking at previous work and theoretical underpinnings first.

rfcs/0000-ret-cont-recursive-nix.md Outdated Show resolved Hide resolved
@Mic92
Copy link
Member

@Mic92 Mic92 commented Feb 6, 2019

So in a nutshell instead of importing nix files from a derivation, this will import derivation files from a derivation? The idea sounds to me the equivalent to autotools/automake, where before a release the configure and Makefile.in files are generated and put in the release tarball.

@edolstra
Copy link
Member

@edolstra edolstra commented Feb 6, 2019

It would be good to add an example and/or some use cases where this approach would apply. For example, I can see it working in the case where you want to build a single derivation (e.g. building Hydra from the release.nix extracted from the Hydra tarball). But it wouldn't address the scenario we have at Tweag where we want to be able to call Bazel from Nix and then have Bazel call Nix to fetch some packages.

Thanks @siddharthist!

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
@Ericson2314
Copy link
Member Author

@Ericson2314 Ericson2314 commented Feb 6, 2019

@Profpatsch

From experience, naïvely recursive systems are usually extremely constrained, so I can see that problem. You should split out the problem with the current proposal from “Motivation” into its own section and start with that.

Yeah, sometime I swear have to open the PR before I can see all the places I dropped in media res to no one's benefit. I'll rewrite things in the next draft to fix that.

Can you frame the idea in terms of already existing systems? Scheme’s call/cc comes to mind, but from the description it is unclear to me whether the ideas are similar. We shouldn’t implement systems like that without looking at previous work and theoretical underpinnings first.

There's a notion of applicative vs monadic build systems. The DRV langauge today is strictly applicative:

data Drv a -- derivation which makes 'a'
type CCompiler = CCode -> Binary
iCanWriteThis :: Drv (CCode -> Binary) -> Drv CCode -> Drv (CCode -> Binary)

This proposal makes it monadic by defining join:

-- | switch to inner after building outer
join :: Drv (Drv a) -> Drv a

Maybe I should make note of this in the RFC :).


Fun fact, operationally, that join is also a lot like Reflex FRP's switchHold. (Ignore the final m, which could be thought of as an ambient interpreter.)

@Ericson2314
Copy link
Member Author

@Ericson2314 Ericson2314 commented Feb 6, 2019

@Mic92 Yes, you can imagine trying to run Nix to the point where there are no drv-producing drvs left, which is a lot like that process those autoconf projects do before release.

@Ericson2314
Copy link
Member Author

@Ericson2314 Ericson2314 commented Feb 6, 2019

@edolstra

It would be good to add an example and/or some use cases where this approach would apply.

Sure, I'll do that with the rephrase @Profpatsch was thinking of.

For example, I can see it working in the case where you want to build a single derivation (e.g. building Hydra from the release.nix extracted from the Hydra tarball).

Indeed it would.

But it wouldn't address the scenario we have at Tweag where we want to be able to call Bazel from Nix and then have Bazel call Nix to fetch some packages.

Well, with this + intentional store, you all would hopefully have no need for Bazel anymore :). But upstream projects will still use it and so it's still worth answering.

With this RFC, there's no reason for Bazel to do any building. Instead, we can have a bazel2nix that will convert the (1st order) Bazel build graph to Nix derivations, which Nix builds as usual. We can (and should) already right this, but currently this tool could only be used "externally", and not within a derivation. With this RFC, using it "internally" is just like instantiating Hydra's release.nix, but with more hoops to jump through first (bazel2nix; nix-instantiate).

(A brief aside: A naive bazel2nix would get GCC for whatever from the PATH, like a configure script, which means we'd have to build GCC first before any bazel2nix step, which reduces parallelism. Much better is to feed bazel2nix GCC's derivation (as a Nix expression or DRV file), so it can just inject that as a dependency in the derivations produced from individual Bazel build nodes that use GCC. Then we get maximal parallelism at the finest granularity, regardless of where the package boundary is!)

Thanks @Mic92

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
rfcs/0000-ret-cont-recursive-nix.md Outdated Show resolved Hide resolved
Thanks @globin

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
@globin
Copy link
Member

@globin globin commented Feb 8, 2019

Per meeting of the @NixOS/rfc-steering-committee this RFC is open for shepherding nominations.

@nixos-discourse
Copy link

@nixos-discourse nixos-discourse commented Feb 8, 2019

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/new-rfcs-40-ret-cont-recursive-nix-open-for-shepherd-nominations/2075/1

@shlevy
Copy link
Member

@shlevy shlevy commented Feb 8, 2019

I'd like to be considered for the shepherd team.

Thanks @siddharthist!

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
@Ericson2314
Copy link
Member Author

@Ericson2314 Ericson2314 commented Feb 10, 2019

@volth Your builtins.exec sounds like import from derivation as it exists today. Is that the case, or am I missing something?

@Ericson2314 Ericson2314 force-pushed the ret-cont branch 2 times, most recently from 71efda4 to c6b6ef9 Feb 10, 2019
@Ericson2314
Copy link
Member Author

@Ericson2314 Ericson2314 commented Dec 6, 2019

That's fine with me. Should we might more about this or ought we to have a prototype implementation / do we even need RFCs for experimental features or only when they become stable.

I suspect the answer is "no", so a lack of meeting on #56 is actually holding me back more than a lack of meeting in this.

@Ericson2314
Copy link
Member Author

@Ericson2314 Ericson2314 commented Jan 29, 2020

Nix finally has some competition with https://www.usenix.org/conference/atc19/presentation/fouladi .

The deal with dynamism exactly as this RFC proposes. In their words "a thunk can write another thunk as its output"; just substitute "thunk" for "derivation", and you have this RFC.

@Ericson2314
Copy link
Member Author

@Ericson2314 Ericson2314 commented Jan 29, 2020

Thanks @volth

@Mic92
Copy link
Member

@Mic92 Mic92 commented Apr 2, 2020

@fpletz did you to create a new meeting for this one?

@Ericson2314
Copy link
Member Author

@Ericson2314 Ericson2314 commented Apr 2, 2020

@Mic92 While I am happy to talk about this again, I was just gong to implement it after I work on CA derivations. The basic idea for both is clear to me, but in doing the programming little intricacies arise that I would never need to anticipate.

I suppose the main thing I'd like to discuss would be a new process RFC ---- getting PRs merged for Nix has been incredibly slow because it is unclear who can review and merge besides @edolstra, and Eelco is busy with many other things. We need a "chain of command" for Nix, deciding who is responsible for reviewing or merging what.

@spacekookie
Copy link
Member

@spacekookie spacekookie commented Apr 16, 2020

@fpletz @edolstra @shlevy @copumpkin Considering there hasn't been any activity on this RFC for a while, maybe you should consider appointing a new shepherd leader for this RFC to get things going again

@Ericson2314
Copy link
Member Author

@Ericson2314 Ericson2314 commented Apr 16, 2020

In addition to what I said above about doing this after CA derivations, there are a few things I want to revise here and an intermediate feature/RFC we could do. Given all that, I am going to close this for now to be reopened when the time is ripe.

@roberth
Copy link
Member

@roberth roberth commented Jul 14, 2021

I like that ret-cont avoids the problem with fixed-output derivations in recursive Nix (as implemented). See NixOS/nix#3205 (comment)

@roberth
Copy link
Member

@roberth roberth commented Sep 17, 2021

This proposal makes it monadic by defining join:

-- | switch to inner after building outer
join :: Drv (Drv a) -> Drv a

I believe it is more like Category and Arrow. Let me explain.

There's no Nixlang Functor for Drv because its type parameter a has a libstore kind, not a libexpr kind. The truly Monadic operation on derivations is import from derivation, which works for any Nixlang expression expr. Its type in the language is import :: forall a :: Expr. Drv a -> a, which, combined with Nix's implicit goal of producing derivations in the end gives the interpreter the ability to Drv (Drv a) -> Drv a. (import shouts Comonad, but let's skip that tangent*)

Ret-cont recursive Nix does not allow IFD. If it were implemented in the language, like RFC 92, it would look like outputOf :: forall a :: StorePath. Drv2 (Drv2 a) -> Drv2 a, where Drv2 is an awkward type that is more about what happens in the sandbox than what happens in the language. What happens in the sandbox is not actually accessible to the language proper, so it is better modeled as an opaque type with some operations that define its behavior, so we get derivation : Build a b where you can think of build as something akin to type Build a b = List a -> Drv b. Now we're really in Arrow territory. We can even define arr by running the evaluator in the sandbox, no wonder there's a separate class hierarchy, because Build just joins the list of cases where arr is not desirable. What we really want to see is (.) of Category, which is Build a b -> Build b c -> Build a c. Applying our hand-wavy type alias definition, we get (List a -> Drv b) -> (List b -> Drv c) -> List a -> Drv c, which after a function application becomes Drv b -> (List b -> Drv c) -> Drv c, so we have to build path b (build : forall p :: Path. Drv p -> p) in order to get at Drv c. This is what ret-cont does, all at the derivation level, not the Nixlang level.

The power spectrum works out the same way. By caller power we have Functor (f b) < Applicative (f b) < Arrow f < Monad (f b) and similarly traditional Nix < ret-cont recursive Nix < import from derivation. Note that callee features are inversely related to caller power. In Haskell, in-language static analysis ends at Arrow (and/or Selective); in Nix build-free evaluation ends at ret-cont recursive Nix*.

*: I'm not including full recursive Nix, I should probably look into Comonad some more and all my steps are too hand-wavy. However, my comment ends here.

@Ericson2314
Copy link
Member Author

@Ericson2314 Ericson2314 commented Oct 16, 2021

@roberth I think a could be the "type" of the derivation outputs. At least with subtyping Drv <= Bytes, Drv (Drv Bytes) -> Drv Bytes makes some sense. But yes it doesn't have the same super-linearizing feel as Monad as used in practice in Haskell, and I do think arrows are underrated if only we'd fix them up like as in https://arxiv.org/abs/1007.2885

@roberth
Copy link
Member

@roberth roberth commented Oct 17, 2021

I think a could be the "type" of the derivation outputs. At least with subtyping Drv <= Bytes, Drv (Drv Bytes) -> Drv Bytes makes some sense.

That's how I read it and it makes total sense at the store level. 👍

Just reading back my comment makes me dizzy though, with three 'languages' to try to keep in my head: derivations, expressions and some sort of meta-language thing consisting of types that we never really defined unambiguously. Pseudo-types as in pseudo-code? Anyway, probably best to just focus on RFC 92.

@Ericson2314
Copy link
Member Author

@Ericson2314 Ericson2314 commented Oct 17, 2021

Yes, thinking about all 3 always got me dizzy too!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment