A simple protocol for derivations to report progress

[copumpkin]

While watching several simultaneous curl downloads in a fixed-output derivation compete for my terminal real estate to report their progress, a thought occurred to me: we all agree that Nix build output is generally too noisy, but we also like to see progress. I love seeing some of the other package managers and their fancy orderly progress reporting.

So how about we do this:

1. Every build already gets a temporary build directory
2. Nix monitors a file with some sort of predefined path inside that folder, perhaps nix-support/progress. In there it should find some predefined format: perhaps <numeric progress out of 100> <short status description if applicable> or something similar. It can either poll or use fancier file watching mechanisms depending on the platform.
3. When nix runs in "orderly mode", it suppresses build stdout/stderr by default (you can get it if the build fails), and instead shows that your derivation is progressing according to to the progress file, updated in an orderly fashion using fancier terminal tricks. If there is no progress file, you just see a notice that the build is happening.
4. Then we go and adjust our various builders (fixed output and otherwise) to emit progress files properly. How complicated that is will depend on the builder, but in general I imagine it won't be terrible.
5. ???
6. Include emoji in our status output
7. Profit

[Ericson2314]

Instead of a directory, could we pass the builder script more pipes beyond the standard streams? That kinda feels cleaner to me though I could be convinced otherwise.

[edolstra]

Indeed a pipe would be more efficient and easier to implement.

[copumpkin]

Sure, I don't feel very strongly about filename, format, or whether it's a pipe or not! I just think having something like this would be helpful.

On May 4, 2016, at 05:20, Eelco Dolstra notifications@github.com wrote:

Indeed a pipe would be more efficient and easier to implement.

―
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub

[vcunat]

Yeah, a pipe, but I see a catch: AFAIK most packages use plain make that does no progress reporting at all. Maybe with a nontrivial patch to our default make...

[copumpkin]

@vcunat the way I was thinking about that: some tools will be easier to plug into progress than others, but we could still express textual progress with make: "configuring -> building -> installing -> testing", and we could reserve a special "I have no clue" value for the progress bar so it doesn't give false precision for cases like that.

The way I envision it:

some-tarball.tgz:      downloading    [########                      ]: 30%
some-makefile-project: configuring...
some-cmake-project:    compiling      [################              ]: 55%

Where fetchurl wrote to its pipe:

50 downloading

The autotools/make project:

? configuring...

The cmake project:

80 compiling

[vcunat]

Older closely related work: https://github.com/qknight/nix-build-view

[vcunat]

BTW, if we monitored the usr+sys time used by each build user, that might be a pretty stable value on comparable HW, not much affected by waiting on I/O, number of threads available, etc. These might be also collected on Hydra so that client could fetch the time estimate.

[Ericson2314]

IMO the data structure should be a tree of fixed-finite progresses, and the protocol a stack. So the generic builder script would do phases-done : num-phases. Then each phase / per-package code can optionally introduce more depth for downloads etc.

[copumpkin]

@Ericson2314 what would that be used for? Are people likely to care about that level of detail? Also, packages don't ever download things in Nix. I'd rather not make it too complicated and get most of the benefit with the least effort.

[Ericson2314]

@copumpkin I guess it doesn't seem to complicated to me :)? Think how often we template together builder script, (generic stuff, haskell and other languages). The template script cannot know the final number of steps but the templating processes is already a "tree of delegations" matching my scheme.

[copumpkin]

Sketch it out in a bit more detail then? It's just not clear to me what types of scenarios this would help with over the basic proposal.

[Ericson2314]

Here's one:

1. So like I said, the generic script can do 4 steps or whatever for the phases
2. The Haskell infra has some idea how many libs/exes/tests are being built per-package
3. The ghc --make output can be parsed for per-module progress per libs/exes/tests

[Ericson2314]

Nix itself can reuse the tree-based logic for combing the progress of multiple derivations being built at a time.

[vcunat]

Also recursive make :-) But I'm not really convinced that allowing nesting would help much to get more precise idea about the progress.

[Ericson2314]

But number of modules * number of binaries * number of phases cannot readily be computed ahead of time.

[Ericson2314]

Now what would be complicated is trying to properly support parallelism. Then the push-pop protocol won't work any more, but I'm happy to kick the bucket on that :).

[copumpkin]

Ultimately it just seems like you'd get 90% of the benefit without nesting,
and that the benefits brought by nesting are ultimately at a level of
detail beyond what someone wants in a progress reporter. If I can get high
LOD that updates 10 times a second in a curses display that doesn't persist
on screen, I don't see much gain, because humans aren't reading the 10
updates per second.

If we maintain the tree after the build to get fine-grained stats about
build times that we can diff in Hydra, I find that a bit more compelling
but I still wouldn't personally put in the effort to do the tree over the
simple "linear" progress report.

I also want to consider the difficulty implementing progress reporting
for individual builders. I want it to be easy for nixpkgs to adopt this.

On Thu, May 5, 2016 at 12:14 John Ericson notifications@github.com wrote:

Now what would be complicated is trying to properly support
parallelism. Then the push-pop protocol won't work any more, but I'm happy
to kick the bucket on that :).

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#896 (comment)

[Ericson2314]

I just don't see how it would work otherwise. What number of steps can we pre-compute that isn't just just the number of phases?

But yeah for hydra, or for more advanced future front-ends, would be where this shine the most.

[copumpkin]

@Ericson2314 CMake already computes its own progress, using whatever mechanisms it has. curl can output progress, as can git and other VCSes. I'm basically saying let individual builders figure out how to deal with their own notions of progress, and we just feed those back to Nix. For things where that's hard, I'm just saying "let's not bother trying to compute a percentage", rather than "let's figure out how to make it work"

[copumpkin]

@Ericson2314 anyway, if you're willing to implement your idea please don't let me dissuade you, but I don't want the perfect to be the enemy of the good here. I personally don't want to be figuring out how to do progress reporting for things that haven't supported it in the past; I'd rather just figure out the simplest way to pass some notion of progress (or even cop out if we can't compute a number) to our users so that we can cut down on the painfully noisy builds that plague Nix today.

[vcunat]

let individual builders figure out how to deal with their own notions of progress

More importantly, this is still extensible – if we figure out some way to measure progress for some class of hierarchical builds, we can plug that layer into those builders.

[Ericson2314]

Yeah I'd be willing to implement this. Ignoring messages for now, pipe protocol represents:

Command ::= Divide n
         |  Checkpoint

For a single status bar, I'd do "rational number interpretation" where the subtree adds to 1 and sibling checkpoints have the same weight. For example Divide 4; Checkpoint; Checkpoint; Divide 3; Checkpoint; Checkpoint; Checkpoint; Checkpoint yields progresses of 0; 1/4; 2/4; 7/12; 8/12; 9/12; 1 (one more fraction than Checkpoint because initial zero).

Might even be easier to do messages "out of band" on a separate pipe --- they need not correspond to checkpoints.

[copumpkin]

Is that logic ultimately just going to emit a fraction? If so, why does it need to be built into Nix as opposed to Nix just consuming the fraction it outputs, and people layering it into their builders, as @vcunat was suggesting?

Like I could still envision Nix consuming a simple "[number or ?] [status]" format, and then some builders pulling in your nested progress helpers (which then live in nixpkgs) if they can support nested progress. Or do you envision Nix actually doing something useful with that nesting information?

[stapelberg]

I think an easy first step would be to:

1. Replace build output with a progress bar on the derivation level, e.g. “3 of 197 derivations built”
2. Buffer and print build output on failure.

Once these are done, granularity can be made finer by introspecting build systems.

[domenkozar]

Just for reference: https://lastlog.de/blog/posts/nix-build-view_using_ncurses.html

[vcunat]

IIRC an important argument was that the UI would better be completely decoupled from nix itself. (Dbus was also suggested as a standard way to publish similar kinds of information.)

[ToxicFrog]

I would really like to see this; in rough order of priority, what I care about is:

• the overall progress (e.g. @stapelberg 's 3 of 197 derivations built)
• the up-front cost (how many derivations to build, total download size, total size on disk) before the build starts
• the progress of each individual build, or at least, some idea of what it's doing (downloading, unpacking, building, installing...)

nix-build-view, based on that screenshot, is exactly what I want -- but I'm using NixOS rather than plain Nix, which means I'm actually running nixos-rebuild and it's hardcoded to call nix-build.

If I have time over the holidays I might have a go at @stapelberg 's idea.

[copumpkin]

dbus would be a non-starter for any non-Linux platform

[taktoa]

Continuing from the analysis I did in #1297 (comment), I've come to the conclusion that Cap'n Proto is probably the right choice for this, since:

1. Cap'n Proto is pretty simple / lightweight / well-thought-out.
2. It offers both an RPC protocol and an IDL/schema language.
3. It's supported on a Linux/Darwin/Windows, and unlike ZeroMQ it uses IOCP on Windows (it is very slow to emulate Unix domain sockets on Windows).
4. It's actively developed (unlike nanomsg which also fixes the IOCP problem that ZeroMQ has).

This may seem like overengineering, since a FIFO could in theory suffice (assuming that there is something like mkfifo on Win32 and that there are no differences between the darwin and linux fifo implementations), but I think that we should leave this open for other kinds of data the build may want to report. Using a proper RPC framework gives us room to change things without having to rip up what we already had.

[copumpkin]

My main concern is adding a fairly meaty dependency to Nix for something that doesn't really need to be very complicated. I'm wary of bringing in a big fancy framework "just in case" someone thinks of something a build might want to report. I mostly think we should stick to the traditional derivation outputs mechanism for most things builds will produce, and then do as little work as we can to add progress to that.

[vcunat]

Pipes: I'm not that sure about speed, but libuv uses "named pipes" on Windows, so there's probably at least one way there. I don't think we need to worry much about the progress-reporting feature on Windows (or even its speed there). The format can probably be made so simple that (de)serialization can be just written by hand.

[copumpkin]

Yeah, we were just talking about simple JSON over a pipe on IRC. Seems like it would cover most common cases and make it fairly easy to extend, while still being shell-friendly.

[copumpkin]

@edolstra I wouldn't mind putting together an attempt at the simplest progress reporting mechanism I can think of. My main concern is a good way to do fancy terminal updates without bringing in new dependencies. Do you have a preference for how to do fancy terminal stuff in Nix today? I guess now that nix-repl is in the main repo you're already using something, right?

[edolstra]

Commit 4af2611 adds support for derivations to use the Nix 1.12 progress mechanism. Thus builtins.fetchurl downloads now show up in the nix build progress bar, e.g.

[8/27/592 built (1 failed), 103.0/302.0 MiB DL] downloading 'http://curl.haxx.se/download/curl-7.53.1.tar.bz2'

A GUI or a full-screen TUI can be implemented by subclassing Logger, see src/nix/progress-bar.cc.

@copumpkin Just ANSI escape codes. nix-repl used readline/ncurses, but nix repl doesn't.

[edolstra]

BTW, builders send progress updates to Nix by writing them to stderr as @nix <json>. We could use a separate pipe for these messages, but that would be a bit more work to implement. I guess it would be cleaner though.

[copumpkin]

That is awesome, thanks! I don't feel super strongly about pipe vs stderr but the latter would reduce potential ambiguity. Only one real comment on the current design:

Adding an optional nesting option (start with a parent ID) would allow us to talk about e.g., "compiling" as a high-level activity, with individual components as sub-activities. It might also improve uniformity of the code, since stepping higher up, you can think of all of an individual build's activities as sub-activities on the build itself.

Otherwise, this looks great!

[edolstra]

@copumpkin I've added nesting (not exposed to builders though, but used to expose the relationship between top-level activities such as substituting and lower-level activities like downloading).

Also, builders can now update the "phase" they're in (0ac35b6). So, for example, in stdenv's genericBuild we could do

            echo "@nix { \"action\": \"setPhase\", \"phase\": \"$curPhase\" }"

[copumpkin]

@edolstra that's awesome!

[copumpkin]

@edolstra are you leaning towards leaving these messages in stderr, or do you think the pipe is the way to go? I don't feel strongly

[Ericson2314]

I would use an additional file descriptor, which avoids anything else getting in the way along with any need for @nix.

edit Oh, I should have scrolled up. Nothing that's not been said before!

[lukego]

How do I see a demo of the progress bar? I have built Nix from the tip of master but I am not sure what commands/options are needed to see the progress messages.

[edolstra]

The progress bar is only enabled for the nix command. So for example

$ nix build nixpkgs.thunderbird
[0/1 built, 8/45/83 copied (75.4/260.0 MiB), 19.9/80.5 MiB DL] fetching freetype-2.6.5 from https://cache.nixos.org

This works only as root because progress messages aren't tunneled over the nix-daemon client connection. (I'm working on that right now.)

I also use --store a lot for testing, to use a temporary store directory:

$ nix build  --store local?root=/tmp/nix nixpkgs.thunderbird  --no-remote-builds 

or with nix copy:

$ nix copy -r --to local?root=/tmp/nix /run/current-system --no-check-sigs 
[8/9/875 copied (40.2/1175.8 MiB)] copying path '/nix/store/c07gdr6cm43j1cphadzafq185k711vx4-coreutils-8.26'

[canndrew]

Any progress on this? Being able to see how many derivations nixos-rebuild has left to build would be extremely helpful.

[lheckemann]

@canndrew this is already essentially possible, just not neatly integrated. On my chromebook, which has to build everything from source (armv7) I usually do this so I have an idea of how long it'll take: just run nix build -f '<nixpkgs/nixos>' system to build the system before running nixos-rebuild.

(edit: shorter equivalent command)

[edolstra]

Closing this since Nix 2 has a progress indication mechanism.