Wishlist: Tup-like

[chenglou]

(Discussed in chat, but I'm making an issue here to track this.)

In the Tup build system, there's this ​_ONE_​ big feature that literally boosted my productivity by like 5x: Tup checks that your deps and targets for the action are really what you claimed them to be. So it'll provide you with some of the most impressive build system error messages I've ever seen, e.g. "you specified that this command outputs to foo.ml, but it actually didn't", "you are about to programmatically write to a file that, in the last round, was manually modified" and "this command read from a file that has not been specified as a dependency". It's the first time I experience "when it build, it works".

If some bash commands/transitive actions' implementation change under the hood, those checks will catch them, while for most build system it'd show up as e.g. "ocamlc failed to compile: etc.".

This way, Tup can also auto include all the (non-generated) dependencies and allow you to over-specify your deps while it checks for the actual more granular deps under the hood.

Lastly, this feature was really helpful to me for learning how e.g. ocamlc worked. I didn't know which files were searched when I include some stuff with -I. Tup generated the graph and told me which files were read.

Of course, Jenga being written in OCaml helped catch many errors already. I initially mentioned this Tup feature in chat because I was very impressed by it, but didn't hope Jenga would gain this feature; apparently, with some docker setup this isn't that hard to achieve, so here we are.

[yminsky]

Can you say more about the mechanisms that tup uses for discovering these
dependencies? That seems like a rather important ingredient...

On Mon, May 16, 2016, 6:14 PM Cheng Lou notifications@github.com wrote:

(Discussed in chat, but I'm making an issue here to track this.)

In the Tup build system, there's this ​_ONE_​ big feature that literally
boosted my productivity by like 5x: Tup checks that your deps and targets
for the action are really what you claimed them to be. So it'll provide you
with some of the most impressive build system error messages I've ever
seen, e.g. "you specified that this command outputs to foo.ml, but it
actually didn't", "you are about to programmatically write to a file that,
in the last round, was manually modified" and "this command read from a
file that has not been specified as a dependency". It's the first time I
experience "when it build, it works".

If some bash commands/transitive actions' implementation change under the
hood, those checks will catch them, while for most build system it'd show
up as e.g. "ocamlc failed to compile: etc.".

This way, Tup can also auto include all the (non-generated) dependencies
and allow you to over-specify your deps while it checks for the actual more
granular deps under the hood.

Lastly, this feature was really helpful to me for learning how e.g. ocamlc
worked. I didn't know which files were searched when I include some stuff
with -I. Tup generated the graph and told me which files were read.

Of course, Jenga being written in OCaml helped catch many errors already.
I initially mentioned this Tup feature in chat because I was very impressed
by it, but didn't hope Jenga would gain this feature; apparently, with some
docker setup this isn't that hard to achieve, so here we are.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#23

[chenglou]

Tup uses FUSE for OS X & Linux under the hood and instruments the file reads/writes. In practice, this has worked really well, except for symlinks and anything absolute path-related. Also, FUSE didn't work correctly on our corporate laptops. For these reasons they were thinking of going back to the old way of doing it, e.g. LD_PRELOAD on Mac, [not sure] on Linux and keep their current DLL injection stuff on windows.

@avsm mentioned "we could do detection with docker commit between rules". I'm not too familiar with docker though.

[Nick-Chapman]

I think the idea of validating dependency correctness by automatically tracking reads and writes made by a command is a cool feature, but very hard to get it right. We thought about such a thing in the early days of jenga, but for one reason or another we never did it, instead trusting the rule writer to get the dependencies.

A similar kind of validation can be done by sandboxing the actions, Jenga has some limited support for this. Option [-sandbox-actions]. We don't select this mode for general building, but it can be useful when new rules are being developers. I'm not sure how robust the current sandbox support is. I believe Leo has been working on it recently.