Failure to actually run the builds in the queue #433
Comments
|
Update: I took a look at these two lines and added the following debug prints: printMsg(lvlError, format("Can this happen!?"));
if (!build->finishedInDB) { // FIXME: can this happen?
printMsg(lvlError, format("YES, THIS CAN HAPPEN"));
(*builds_)[build->id] = build;
}
I'm glad to be of service and answer the question in the comment, but I'm not yet sure what it means. This is the contents of the |
|
I see the exact same problem on our hydra instance. Hydra runs on and builds for a normal intel 64bit machine, no cross-compilation is involved. We are using Hydra version de55303 . |
|
I'm in a similar situation and currently I suspect that the problem is the |
|
1.11.7 |
|
@FPtje the problem that I had, has a much simpler explanation. The |
|
Due to a tip from @expipiplus1 I managed to fix our long standing issue with hydra not dequeueing certain jobs. Note that I have always configured hydra as follows: { services.hydra.buildMachinesFiles = mkIf (config.nix.buildMachines == []) []; }If you don't explicitly set In our case we don't set I suspect that the jobs that fail to dequeue have dependencies with certain The solution is to explicitly specify {
nix.buildMachines = [
{ hostName = "localhost";
system = "x86_64-linux";
supportedFeatures = ["kvm" "nixos-test" "big-parallel" "benchmark"];
maxJobs = 8;
}
];
}We should probably document this behaviour in the description of the |
Problem description
Our private hydra build server (with @basvandijk) has had its database cleaned and currently holds one project with one jobset containing one (fresh) job. This job is put inside the Queue (as seen in
Status > Queue), but it is never actually built.When looking at the job's
Summarypage, theStatusshowsScheduled to be built, part of evaluation 1. The build steps tab is empty. The build dependencies tab, though, shows a whole bunch of dependencies.The
Status > Latest stepspage is empty, as is theStatus > Latest builds. TheStatus > Running buildssays there are no running builds. The machines inMachine statusare shown as Idle.Background
The issue has started since hydra was updated a month or so ago. In a debugging session (together with @basvandijk), the version was upgraded to revision
de55303197d997c4fc5, where the issue still occurs. Jobs just wouldn't build. Only when actually runningnix-store --realize <drv path>does hydra mark the job as finished.Somewhat related to this issue is #431, in which @basvandijk describes some error with a Raspberry Pi 3 build server. He thought the Raspberry Pi stuff might cause Hydra to act this weird, but the one job currently active has nothing to do with the Raspberry Pi, which should exclude it as a cause.
Debug information
With all
lvlChatprint messages changed tolvlInfo(because we're too lazy to figure out how to set verbose mode), the following is printed in the journal when the configuration is switched (and hydra is started):Subsequently, this is what hydra tells us every now and then afterwards:
With some debug messages I figured out that
hydra-queue-runnerblocks on this line after mentioning that those 77 steps are "now runnable". From that point it never unblocks until the process is manually restarted.So somehow the steps are marked as "runnable", but nothing is actually built. Restarting
hydradoesn't change anything and just reproduces the exact same output as posted above.Further debugging
I'm actually kind of lost on ideas as to what could cause this. I'll update the issue with more info as I continue, but where should I even look?
In my current setup it's easy to put debug prints in arbitrary places in the code of hydra, if necessary.
The text was updated successfully, but these errors were encountered: