Build a lot more stuff

[grahamc]

We've got a good bit of build capacity and it goes largely unused:

Worse, x86_64-linux has effectively unbounded capacity due to a generous spot instance backing. Darwin is the least strong, and aarch64-linux is in the middle.

For these reasons, I'd like to build way more PRs and use this capacity to make merging PRs easier and safe.

Phase One: Simple Expansion

• Setup sandboxing properly on all the macs
• Backport the sandboxing code from master to 17.09
• Expand caller access to everyone with merge writes on nixos/nixpkgs

Phase Two: Easier Build Selection

• Implement build "sampling" where ofborg automatically selects a handful of attrs to build based on what has changed in the PR.
• Implement test sampling where ofborg does the same thing, but selects a handful of changed tests.

Phase Three: Limited Automatic Building

• Sucker Convince someone to buy me / provide me with a bunch of macs to enroll for capacity
• Automatically sample PRs from anyone trusted to call ofborg

Phase Four: Automatic Building

• Automatically sample all PRs

[zimbatm]

I asked @macstadium if they could provide some build capacity, no answer so far

[7c6f434c]

So, how many macOS builders there are now? Maybe (temporarily) allow to separately ask borg to build just i686-linux and x86_64-linux, to emphasize abundance of build capacity on this platform?

[grahamc]

Right now we have three mac builders. I think that is a good idea, @7c6f434c, and solves the issue around darwin not having adequate sandboxing for this use case. I am planning on the following:

Users bucketed by the following:

• trusted: explicitly trusted by the ofborg configuration
• known: has merge rights to nixpkgs
• unknown: everyone else

1. if you are a trusted user, the build/test job is sent to all architectures.
2. if you are a known user, the build/test job is only sent to architectures which has adequate sandboxing (aarch64-linux, x86_64-linux).
3. if you are unknown, the command is ignored (as it is now)

With the addition of this change, all known members will be allowed to trigger the eval command.

[7c6f434c]

Erm. Doesn't i686-linux have adequate sandboxing? Aren't all its builders actually x86_64-linux?

Also, given the capacity skew, I think a trusted user might still want to opt out from sending to darwin?

[grahamc]

Let's wait and see if we actually have capacity issues on mac before scaling it back for trusted users. Ofborg doesn't run anything on i686.

[LnL7]

I agree, we can monitor the queue and revisit if necessary. At the moment we actually have the same number of builders for every platform.

[7c6f434c]

Let's wait and see if we actually have capacity issues on mac before scaling it back for trusted users. Ofborg doesn't run anything on i686.

I guess what you meant is «before spending effort on cofigurability» (my proposal gives the trusted users an option to scale back, but allows them full runs, too). Fair enough, though.

[grahamc]

A simplified way to sample PRs. During the evaluation step:

• parse commit messages for the subject package (ie: "postfix: foo bar baz" -> postfix)
• if the package is a buildable attribute according to the evaluation, schedule a build job for it

If the user is trusted, send it to all platforms. If the user is known, send it to just platforms with good sandboxing. If the user is unknown, don't schedule any jobs.

This comes with the benefit of encouraging precisely following of the commit format. For example, knowing your package will auto-build may be the difference between:

grv: init at 0.1.0

and:

gitAndTools.grv: init at 0.1.0

[andir]

You might not always want to build it or at least have a blacklist? I think building chromium would waste a lot of cpu cycles with probably low benefit?

[7c6f434c]

This comes with the benefit of encouraging precisely following of the commit format. > gitAndTools.grv: init at 0.1.0

There are some fixes that simultaneously fix two or three packages and logically should be committed together; maybe support splitting by «,»?

[7c6f434c]

In the ideal world, Chromium would be built on the platforms with proper sandboxing, except those where the submitter has tried to build before submitting. (In the ideal world everyone uses the best sandboxing available for tests)

[7c6f434c]

By the way, judging from the acoustic evidence, there is significant progress on the original goal of builder utilisation. At least for some combinations of day-of-the-week and hour-of-the-day.

[wmertens]

How about building all dependencies for a given PR?

[7c6f434c]

@wmertens I think ofBorg currently doesn't allow to build just Chromium alone (timeout will happen sooner). What you say is way above capacity right now.

[wmertens]

aha ok :) other question: how about running headless darwin in VMs? That way would solve Darwin capacity, right? Another option might be cross-compiling but Nixpkgs is not there yet…

…

On Fri, Mar 23, 2018, 6:17 PM Michael Raskin, ***@***.***> wrote: @wmertens <https://github.com/wmertens> I think ofBorg currently doesn't allow to build just Chromium alone (timeout will happen sooner). What you say is way above capacity right now. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#30 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADWlsKPRs3vsGhZzAKokpdH5QdmbQnmks5thS4zgaJpZM4RMW4f> .

[7c6f434c]

I don't think we have even x86_64-linux capacity. Is there any Darwin option that can be legally ran on non-Apple hardware and is close enough to macOS for our build needs?

[wmertens]

hmm, legally :-/ there is https://github.com/PureDarwin/PureDarwin but it doesn't look like it actually works…

…

On Fri, Mar 23, 2018, 9:00 PM Michael Raskin, ***@***.***> wrote: I don't think we have even x86_64-linux capacity. Is there any Darwin option that can be legally ran on non-Apple hardware and is close enough to macOS for our build needs? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#30 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADWlgSoJPAO96b9SMyI9LIRz_qQQZ-Jks5thVR2gaJpZM4RMW4f> .

[7c6f434c]

It might even work per se, but Nixpkgs seems to depend on some Apple frameworks…

Anyway, we don't even have enough capacity for crazy megaprojects on the Linux side.

[7c6f434c]

I think some checkboxes here are either obsolete or done…