Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't distribute localhost builds to other builders #749

Merged
merged 1 commit into from May 4, 2020

Conversation

@lopsided98
Copy link
Contributor

lopsided98 commented May 3, 2020

Since the recent Hydra upgrade in nixpkgs, I have noticed that builds that are supposed to be running on localhost are actually getting distributed to other machines configured in /etc/nix/machines. If the build fails on the remote machine, it is marked as aborted.

I suspect this is caused by a change in Nix, rather than in Hydra, but I couldn't identify a commit that is to blame. It does not seem to happen for builds that Hydra intentionally runs on remote machines, even if those remote machines have builders configured in /etc/nix/machines. This may be due to implementation differences between the cmdBuildPaths nix-store --serve command (used for localhost builds) and the cmdBuildDerivation command (used for remote builds).

Note that I am still running hydra-migration because I was unable to get the migration to complete successfully.

This PR passes --builders '' to nix-store to prevent it from using Nix's remote building system.

@edolstra edolstra merged commit ace30b4 into NixOS:master May 4, 2020
1 check passed
1 check passed
tests
Details
@lopsided98 lopsided98 deleted the lopsided98:localhost-no-remote branch May 4, 2020
@basvandijk
Copy link
Member

basvandijk commented May 10, 2020

I also noticed our localhost builder building on remote builders. After this commit those remote builds issued by the localhost builder are gone. So far so good...

However, I now see another issue: At the moment our Hydra is building the following preferLocalBuild derivation on the localhost builder:

[root@hydra:/]# nix show-derivation /nix/store/fn433i69vkgx9rilkj5iwc9znsv6ddqy-unit-minica.service.drv | grep preferLocalBuild
      "preferLocalBuild": "1",

The problem is it is not just building this derivation it's also building all of its dependencies locally!

Why are these dependencies not first scheduled to my remote builders before only building this final preferLocalBuild derivation on my localhost builder?

Note I configured the localhost builder on Hydra like this:

{
      nix.buildMachines = [
        {
          hostName = "localhost";
          system = "x86_64-linux";
          maxJobs = 4;
          mandatoryFeatures = [ "local" ];
          speedFactor = 4;
        }
      ]
      ++ ... various remote builders ...;
}
@AmineChikhaoui
Copy link
Member

AmineChikhaoui commented May 10, 2020

@basvandijk that was an old bug, I had to revert d4b4255#diff-09d35108797d5e4cc3e05fb040546314 to have it work correctly.

@basvandijk
Copy link
Member

basvandijk commented May 10, 2020

My suspicion is that even though I have configured hydra with

services.hydra.useSubstitutes = true;

steps build by the localhost builder are not utilizing my configured binary caches and thus have to build all already built dependencies locally.

I'm not sure yet how to confirm my hypothesis... @edolstra any ideas?

@basvandijk
Copy link
Member

basvandijk commented May 10, 2020

@AmineChikhaoui thanks! I'm going to see if reverting d4b4255 fixes it...

@basvandijk
Copy link
Member

basvandijk commented May 11, 2020

I can confirm that reverting d4b4255 fixes it. Now only preferLocalBuild derivations are build on my localhost builder.

basvandijk added a commit to basvandijk/hydra that referenced this pull request May 11, 2020
This reverts commit d4b4255.

That commit caused builds on the localhost builder to build all
dependencies of `preferLocalBuild` derivations as well. It was
[suggested](NixOS#749 (comment))
to revert that commit to fix this and I can confirm it indeed fixes it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.