New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC 0040]: "Ret-cont" recursive Nix #40

Open
wants to merge 12 commits into
base: master
from

Conversation

Projects
None yet
10 participants
@Ericson2314
Copy link
Member

Ericson2314 commented Feb 6, 2019

"Ret-cont" recursive Nix is a restricted form of recursive Nix, where one builds a derivations instead of executing builds during the builds. This avoids some platform-specific contortions relating to nested sandboxing. More importantly, it prevents imperative and overly linear direct-style build scripts; easy to write but throwing away the benefits of Nix.

Rendered

Ericson2314 added some commits Feb 1, 2019

@Ericson2314 Ericson2314 changed the title "Ret-cont" recursive Nix [RFC 0040]: "Ret-cont" recursive Nix Feb 6, 2019

@Profpatsch

This comment has been minimized.

Copy link
Member

Profpatsch commented Feb 6, 2019

I don’t entirely understand the proposal.

From experience, naïvely recursive systems are usually extremely constrained, so I can see that problem. You should split out the problem with the current proposal from “Motivation” into its own section and start with that.

Can you frame the idea in terms of already existing systems? Scheme’s call/cc comes to mind, but from the description it is unclear to me whether the ideas are similar. We shouldn’t implement systems like that without looking at previous work and theoretical underpinnings first.

@Mic92

This comment has been minimized.

Copy link

Mic92 commented Feb 6, 2019

So in a nutshell instead of importing nix files from a derivation, this will import derivation files from a derivation? The idea sounds to me the equivalent to autotools/automake, where before a release the configure and Makefile.in files are generated and put in the release tarball.

@edolstra

This comment has been minimized.

Copy link
Member

edolstra commented Feb 6, 2019

It would be good to add an example and/or some use cases where this approach would apply. For example, I can see it working in the case where you want to build a single derivation (e.g. building Hydra from the release.nix extracted from the Hydra tarball). But it wouldn't address the scenario we have at Tweag where we want to be able to call Bazel from Nix and then have Bazel call Nix to fetch some packages.

ret-cont-recursive-nix: Fix typo
Thanks @siddharthist!

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
@Ericson2314

This comment has been minimized.

Copy link
Member Author

Ericson2314 commented Feb 6, 2019

@Profpatsch

From experience, naïvely recursive systems are usually extremely constrained, so I can see that problem. You should split out the problem with the current proposal from “Motivation” into its own section and start with that.

Yeah, sometime I swear have to open the PR before I can see all the places I dropped in media res to no one's benefit. I'll rewrite things in the next draft to fix that.

Can you frame the idea in terms of already existing systems? Scheme’s call/cc comes to mind, but from the description it is unclear to me whether the ideas are similar. We shouldn’t implement systems like that without looking at previous work and theoretical underpinnings first.

There's a notion of applicative vs monadic build systems. The DRV langauge today is strictly applicative:

data Drv a -- derivation which makes 'a'
type CCompiler = CCode -> Binary
iCanWriteThis :: Drv (CCode -> Binary) -> Drv CCode -> Drv (CCode -> Binary)

This proposal makes it monadic by defining join:

-- | switch to inner after building outer
join :: Drv (Drv a) -> Drv a

Maybe I should make note of this in the RFC :).


Fun fact, operationally, that join is also a lot like Reflex FRP's switchHold. (Ignore the final m, which could be thought of as an ambient interpreter.)

@Ericson2314

This comment has been minimized.

Copy link
Member Author

Ericson2314 commented Feb 6, 2019

@Mic92 Yes, you can imagine trying to run Nix to the point where there are no drv-producing drvs left, which is a lot like that process those autoconf projects do before release.

@Ericson2314

This comment has been minimized.

Copy link
Member Author

Ericson2314 commented Feb 6, 2019

@edolstra

It would be good to add an example and/or some use cases where this approach would apply.

Sure, I'll do that with the rephrase @Profpatsch was thinking of.

For example, I can see it working in the case where you want to build a single derivation (e.g. building Hydra from the release.nix extracted from the Hydra tarball).

Indeed it would.

But it wouldn't address the scenario we have at Tweag where we want to be able to call Bazel from Nix and then have Bazel call Nix to fetch some packages.

Well, with this + intentional store, you all would hopefully have no need for Bazel anymore :). But upstream projects will still use it and so it's still worth answering.

With this RFC, there's no reason for Bazel to do any building. Instead, we can have a bazel2nix that will convert the (1st order) Bazel build graph to Nix derivations, which Nix builds as usual. We can (and should) already right this, but currently this tool could only be used "externally", and not within a derivation. With this RFC, using it "internally" is just like instantiating Hydra's release.nix, but with more hoops to jump through first (bazel2nix; nix-instantiate).

(A brief aside: A naive bazel2nix would get GCC for whatever from the PATH, like a configure script, which means we'd have to build GCC first before any bazel2nix step, which reduces parallelism. Much better is to feed bazel2nix GCC's derivation (as a Nix expression or DRV file), so it can just inject that as a dependency in the derivations produced from individual Bazel build nodes that use GCC. Then we get maximal parallelism at the finest granularity, regardless of where the package boundary is!)

ret-cont-recursive-nix: Fix typo
Thanks @Mic92

Co-Authored-By: Ericson2314 <git@JohnEricson.me>

@globin globin added the status:new label Feb 7, 2019

ret-cont-recursive-nix: Fix typo
Thanks @globin

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
@globin

This comment has been minimized.

Copy link
Member

globin commented Feb 8, 2019

Per meeting of the @NixOS/rfc-steering-committee this RFC is open for shepherding nominations.

@nixos-discourse

This comment has been minimized.

Copy link

nixos-discourse commented Feb 8, 2019

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/new-rfcs-40-ret-cont-recursive-nix-open-for-shepherd-nominations/2075/1

@shlevy

This comment has been minimized.

Copy link
Member

shlevy commented Feb 8, 2019

I'd like to be considered for the shepherd team.

ret-cont-recursive-nix: Fix typo
Thanks @siddharthist!

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
@volth

This comment has been minimized.

Copy link

volth commented Feb 9, 2019

To the list of alternatives: builtins.exec producing either derivations or vendored .nix files (Pros: easy and straighforward; no problem with nested sandboxing; Drawbacks: not frienly to parallel and remove builds)

@Ericson2314

This comment has been minimized.

Copy link
Member Author

Ericson2314 commented Feb 10, 2019

@volth Your builtins.exec sounds like import from derivation as it exists today. Is that the case, or am I missing something?

@Ericson2314 Ericson2314 force-pushed the Ericson2314:ret-cont branch 2 times, most recently from 71efda4 to c6b6ef9 Feb 10, 2019

@Ericson2314 Ericson2314 force-pushed the Ericson2314:ret-cont branch from c6b6ef9 to 5564fdb Feb 10, 2019

@Ericson2314

This comment has been minimized.

Copy link
Member Author

Ericson2314 commented Feb 10, 2019

I rewrote the motivation, adding examples as proposed. Next I'll expand the detailed design with some formality.

@volth

This comment has been minimized.

Copy link

volth commented Feb 10, 2019

@volth Your builtins.exec sounds like import from derivation as it exists today. Is that the case, or am I missing something?

The main difference is the "build" code executed as user, not as "nixbld1" and so never in the sandbox

@Ericson2314

This comment has been minimized.

Copy link
Member Author

Ericson2314 commented Feb 11, 2019

@volth Oh right. I forgot that it already exists. I barely consider it an alternative since it's wildly impure and has all the same eval-lengthening issues as import-from-derivation, but I'll add it anyways.

@Ericson2314

This comment has been minimized.

Copy link
Member Author

Ericson2314 commented Feb 11, 2019

Also, the detailed design is rewritten with some half-baked formalism. I'm sure it could be improved, but it hopefully it already helps.

Do a lousy job formalizing the detailed design
Break off some previously inline observations into their own subsection.

@Ericson2314 Ericson2314 force-pushed the Ericson2314:ret-cont branch from 710fec0 to 22f8322 Feb 11, 2019

@shlevy
Copy link
Member

shlevy left a comment

So in general I'm in favor of this idea, and indeed when I implemented recursive nix a few years back I came to and implemented this same general requirement (I called it monadic derivations, though that's sloppy of course). I'm not a huge fan of dumping a sub-store into an output though when we have a perfectly working store available already, and in general I think we're going to need both this and general recursive nix so we should come up with a design that iterates in that direction.

Also, this isn't a major point but I think you should think through a bit more what the nix-lang interface to this functionality will look like.

Let me know if you want to pair up on iterating the design!

Derivations that that do so indicate this with some special attribute, say `__recursive`.
Such derivations must have two outputs, `store` and `drv`.
`store` would be a local Nix store limited to just drvs and fixed output builds.
`drv` would contain a symlink to one of the derivations in the store, the root.

This comment has been minimized.

@shlevy

shlevy Feb 11, 2019

Member

Note that ideally for recursive drvs you also have the actual outputs of the final derivation specified, so the nix-lang usage of them can just reference outputs appropriately

This comment has been minimized.

@Ericson2314

Ericson2314 Feb 11, 2019

Author Member

Ah yes that's true.

where the relationship between the derivation and build can only be verified by redoing the build,
and where even then there's no way to know whether to blame the output for being actually malicious, or the derivation for merely being non-deterministic.

- The substitution of drvs in a downstream derivation reminds me of the substitution of drvs for content hashes with the intensional store.

This comment has been minimized.

@shlevy

shlevy Feb 11, 2019

Member

Yes, this was one of my comments on that PR. We should figure out how to handle reductions more generally: In the continuation-based recursive nix case, we have foo.drv → foo'.drv whereas in the intensional store case we have (foo.drv)!out → cashash-foo, but the principle is the same.

All data can be stuck inside the drv file, so this can be cut without limiting expressive power.
But this is much less efficient, and more cumbersome for whatever produces the data.

- Use a socket to talk to the host daemon.

This comment has been minimized.

@shlevy

shlevy Feb 11, 2019

Member

I'm hugely in favor of this approach. Using nested stores and all that is kind of hacky, and we have lots of reasons besides actually nested builds to want recursive nix (e.g. store querying, dynamic dependency fetching, etc.). In general I'm in favor of moving toward a much more structured way for builds that are nix-aware to talk to the store; builds that aren't can then use generic functionality built on top of that.

This comment has been minimized.

@Ericson2314

Ericson2314 Feb 13, 2019

Author Member

Yeah I don't really care soooo much on this point. I guess I am more interested in pointing out that we can skip the daemon socket, as proof of the simpler computational model that is this version, than insisting that we must skip the daemon socket.

Ericson2314 and others added some commits Feb 11, 2019

ret-cont-recursive-nix: Fix typo
Thanks @Mic92!

Co-Authored-By: Ericson2314 <git@JohnEricson.me>
@Ericson2314

This comment has been minimized.

Copy link
Member Author

Ericson2314 commented Feb 11, 2019

Yes I should have put somewhere that no variation of recursive Nix is truly new, @shlevy has implemented them all at some point :D.

@shlevy

This comment has been minimized.

Copy link
Member

shlevy commented Feb 21, 2019

Eelco has been nominated and accepted the nomination. @dtzWill @copumpkin are you guys game for joining as well?

Instead, we can produce a derivation which has the callee as a dependency, and continuation drv downstream depending on it.
Since the outer derivation evaluates (builds) the inner derivation rather than calling anything, I deem that it returns the derivation.
This gives the "return" part of the name.
Both differences together, the first example becomes something like:

This comment has been minimized.

@ocharles

ocharles Feb 21, 2019

Member

But the first example had two outputs - this one has only one. Does this mean I can only recurse once?

This comment has been minimized.

@Ericson2314

Ericson2314 Feb 21, 2019

Author Member

You can recurse as many times as you want. That one output is the derivation itself, not some output of it. After you've been rewritten into the final derivation (i.e. a non-recursive one), you produce all of its outputs.

This comment has been minimized.

@ocharles

ocharles Feb 21, 2019

Member

Ok, I suppose my question more meant: can you provide behaviour equivalent to your first example. If so, could that be demonstrated in this RFC?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment