Machine status shows every machine twice

[mweinelt]

Describe the bug
In the machine status we see every machine twice.

• Once with its supported architectures

  elated-minsky (x86_64-linux, i686-linux)

• Once with its steps, but the name is displayed with a strikethrough

  elated-minsky (39994 steps done, 32.9 s/step)

Both entries list the builds, so the page is effectively twice as long as it should be.

To Reproduce
Steps to reproduce the behavior:

1. Go to https://hydra.nixos.org/machines

Expected behavior
Every machine should only be listed once with its aggregated metadata.

Screenshots

Hydra Server:

Please fill out this data as well as you can, but don't worry if you can't -- just do your best.

• OS and version: NixOS 24.11 (db8f4fe18ce772a9c8f3adf321416981c8fe9371)
• Version of Hydra: 1c52c4c
• Version of Nix Hydra is built against: nix follows input 'hydra/nix'
• Version of the Nix daemon: 2.28.2

Additional context
I saw this issue on my private hydra every once in a while, so I guess it is flaky behaviour.

[mweinelt]

The queue-runner status does not show this behaviour.

   "machines" : {
      "ssh://build@elated-minsky.builder.nixos.org" : {
         "avgStepBuildTime" : 32.9726295471191,
         "avgStepTime" : 52.5759201049805,
         "consecutiveFailures" : 0,
         "currentJobs" : 3,
         "disabledUntil" : 0,
         "enabled" : true,
         "lastFailure" : 0,
         "mandatoryFeatures" : [],
         "nrStepsDone" : 40009,
         "supportedFeatures" : [
            "benchmark",
            "kvm",
            "nixos-test"
         ],
         "systemTypes" : [
            "i686-linux",
            "x86_64-linux"
         ],
         "totalStepBuildTime" : 1319202,
         "totalStepTime" : 2103510
      },
      "ssh://build@goofy-hopcroft.builder.nixos.org" : {
         "avgStepBuildTime" : 45.7870483398438,
         "avgStepTime" : 68.8115997314453,
         "consecutiveFailures" : 0,
         "currentJobs" : 4,
         "disabledUntil" : 0,
         "enabled" : true,
         "lastFailure" : 0,
         "mandatoryFeatures" : [],
         "nrStepsDone" : 41019,
         "supportedFeatures" : [
            "kvm",
            "nixos-test"
         ],
         "systemTypes" : [
            "aarch64-linux"
         ],
         "totalStepBuildTime" : 1878139,
         "totalStepTime" : 2822583
      },
      "ssh://build@sleepy-brown.builder.nixos.org" : {
         "avgStepBuildTime" : 34.0310134887695,
         "avgStepTime" : 58.3921203613281,
         "consecutiveFailures" : 0,
         "currentJobs" : 4,
         "disabledUntil" : 0,
         "enabled" : true,
         "lastFailure" : 0,
         "mandatoryFeatures" : [],
         "nrStepsDone" : 11318,
         "supportedFeatures" : [
            "big-parallel",
            "kvm",
            "nixos-test"
         ],
         "systemTypes" : [
            "i686-linux",
            "x86_64-linux"
         ],
         "totalStepBuildTime" : 385163,
         "totalStepTime" : 660882
      },
      "ssh://root@135.181.230.86" : {
         "avgStepBuildTime" : 90.8903350830078,
         "avgStepTime" : 132.007232666016,
         "consecutiveFailures" : 0,
         "currentJobs" : 1,
         "disabledUntil" : 0,
         "enabled" : true,
         "lastFailure" : 0,
         "mandatoryFeatures" : [],
         "nrStepsDone" : 2763,
         "supportedFeatures" : [
            "big-parallel",
            "kvm",
            "nixos-test"
         ],
         "systemTypes" : [
            "aarch64-linux"
         ],
         "totalStepBuildTime" : 251130,
         "totalStepTime" : 364736
      },
      "ssh://root@eager-heisenberg.mac.nixos.org" : {
         "avgStepBuildTime" : 121.226058959961,
         "avgStepTime" : 142.68766784668,
         "consecutiveFailures" : 0,
         "currentJobs" : 1,
         "disabledUntil" : 0,
         "enabled" : true,
         "lastFailure" : 0,
         "mandatoryFeatures" : [],
         "nrStepsDone" : 2318,
         "supportedFeatures" : [
            "big-parallel"
         ],
         "systemTypes" : [
            "aarch64-darwin",
            "x86_64-darwin"
         ],
         "totalStepBuildTime" : 281002,
         "totalStepTime" : 330750
      },
      "ssh://root@enormous-catfish.mac.nixos.org" : {
         "avgStepBuildTime" : 135.887222290039,
         "avgStepTime" : 156.23161315918,
         "consecutiveFailures" : 0,
         "currentJobs" : 0,
         "disabledUntil" : 0,
         "enabled" : true,
         "idleSince" : 1744555557,
         "lastFailure" : 0,
         "mandatoryFeatures" : [],
         "nrStepsDone" : 4132,
         "supportedFeatures" : [],
         "systemTypes" : [
            "aarch64-darwin",
            "x86_64-darwin"
         ],
         "totalStepBuildTime" : 561486,
         "totalStepTime" : 645549
      },
      "ssh://root@growing-jennet.mac.nixos.org" : {
         "avgStepBuildTime" : 192.535934448242,
         "avgStepTime" : 213.574035644531,
         "consecutiveFailures" : 0,
         "currentJobs" : 1,
         "disabledUntil" : 0,
         "enabled" : true,
         "lastFailure" : 0,
         "mandatoryFeatures" : [],
         "nrStepsDone" : 3019,
         "supportedFeatures" : [],
         "systemTypes" : [
            "aarch64-darwin",
            "x86_64-darwin"
         ],
         "totalStepBuildTime" : 581266,
         "totalStepTime" : 644780
      },
      "ssh://root@intense-heron.mac.nixos.org" : {
         "avgStepBuildTime" : 133.94612121582,
         "avgStepTime" : 149.6201171875,
         "consecutiveFailures" : 0,
         "currentJobs" : 0,
         "disabledUntil" : 0,
         "enabled" : true,
         "idleSince" : 1744555957,
         "lastFailure" : 0,
         "mandatoryFeatures" : [],
         "nrStepsDone" : 4325,
         "supportedFeatures" : [],
         "systemTypes" : [
            "aarch64-darwin",
            "x86_64-darwin"
         ],
         "totalStepBuildTime" : 579317,
         "totalStepTime" : 647107
      },
      "ssh://root@kind-lumiere.mac.nixos.org" : {
         "avgStepBuildTime" : 135.507843017578,
         "avgStepTime" : 160.006652832031,
         "consecutiveFailures" : 0,
         "currentJobs" : 1,
         "disabledUntil" : 1744497467,
         "enabled" : true,
         "lastFailure" : 1744497389,
         "mandatoryFeatures" : [],
         "nrStepsDone" : 2103,
         "supportedFeatures" : [
            "big-parallel"
         ],
         "systemTypes" : [
            "aarch64-darwin",
            "x86_64-darwin"
         ],
         "totalStepBuildTime" : 284973,
         "totalStepTime" : 336494
      },
      "ssh://root@maximum-snail.mac.nixos.org" : {
         "avgStepBuildTime" : 155.535369873047,
         "avgStepTime" : 174.150238037109,
         "consecutiveFailures" : 0,
         "currentJobs" : 0,
         "disabledUntil" : 0,
         "enabled" : true,
         "idleSince" : 1744555491,
         "lastFailure" : 0,
         "mandatoryFeatures" : [],
         "nrStepsDone" : 3661,
         "supportedFeatures" : [],
         "systemTypes" : [
            "aarch64-darwin",
            "x86_64-darwin"
         ],
         "totalStepBuildTime" : 569415,
         "totalStepTime" : 637564
      },
      "ssh://root@sweeping-filly.mac.nixos.org" : {
         "avgStepBuildTime" : 189.778945922852,
         "avgStepTime" : 205.381317138672,
         "consecutiveFailures" : 0,
         "currentJobs" : 0,
         "disabledUntil" : 0,
         "enabled" : true,
         "idleSince" : 1744555492,
         "lastFailure" : 0,
         "mandatoryFeatures" : [],
         "nrStepsDone" : 3126,
         "supportedFeatures" : [],
         "systemTypes" : [
            "aarch64-darwin",
            "x86_64-darwin"
         ],
         "totalStepBuildTime" : 593249,
         "totalStepTime" : 642022
      },
      "ssh://root@t4b.cunat.cz" : {
         "avgStepBuildTime" : 25.0773067474365,
         "avgStepTime" : 38.0972557067871,
         "consecutiveFailures" : 0,
         "currentJobs" : 5,
         "disabledUntil" : 0,
         "enabled" : true,
         "lastFailure" : 0,
         "mandatoryFeatures" : [],
         "nrStepsDone" : 401,
         "supportedFeatures" : [
            "kvm",
            "nixos-test"
         ],
         "systemTypes" : [
            "x86_64-linux"
         ],
         "totalStepBuildTime" : 10056,
         "totalStepTime" : 15277
      }
   },

[mweinelt]

Or thrice.

hopeful-rivest (aarch64-linux)
hopeful-rivest (46 steps done, 180.4 s/step)
hopeful-rivest

[Mic92]

This is the query we are using for this:

    $c->stash->{machines} = $machines;
    $c->stash->{steps} = dbh($c)->selectall_arrayref(
        "select build, stepnr, s.system as system, s.drvpath as drvpath, machine, s.starttime as starttime, jobsets.project as project, jobsets.name as jobset, job, s.busy as busy " .
        "from BuildSteps s " .
        "join Builds b on s.build = b.id " .
        "join Jobsets jobsets on jobsets.id = b.jobset_id " .
        "where busy != 0 order by machine, stepnr",
        { Slice => {} });

[Mic92]

Also machines actually come from here:

hydra/src/lib/Hydra/Helper/Nix.pm

Line 341 in bdde73a

sub getMachines {

[Mic92]

The latter function uses a hashmap so duplicates would actually be removed by the latest machine wins approach.

[Mic92]

I wonder if there is any javascript in the front-end that could somehow append the dom twice to cause this? The perl code looks like it wouldn't allow this.

[mweinelt]

❯ curl https://hydra.nixos.org/machines | grep elated
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0          <tt >elated-minsky</tt>            <span class="muted" style="font-weight: normal;">
          <tt style="text-decoration: line-through;">elated-minsky</tt>            <span class="muted" style="font-weight: normal;">
100  126k  100  126k    0     0   249k      0 --:--:-- --:--:-- --:--:--  249k

This is absolutely in the rendered template.