Nixpkgs config not propagated (nixpkgs.system, possibly more)

[justinas]

I have an aarch64-linux machine in my hive. Recently updated to master of colmena and it tries to build for x86_64-linux instead, resulting in a cryptic error as it tries to execute x86_64 perl/bash:

[ERROR] Deployment to robotaki failed. Last 8 lines of logs:
[ERROR] /nix/store/2l0hln4hzpqv65r2wamkxd9gcvslg6kv-nixos-system-robotaki-21.05pre-git/bin/switch-to-configuration: line 3: use: command not found
[ERROR] /nix/store/2l0hln4hzpqv65r2wamkxd9gcvslg6kv-nixos-system-robotaki-21.05pre-git/bin/switch-to-configuration: line 4: use: command not found
[ERROR] /nix/store/2l0hln4hzpqv65r2wamkxd9gcvslg6kv-nixos-system-robotaki-21.05pre-git/bin/switch-to-configuration: line 5: use: command not found
[ERROR] /nix/store/2l0hln4hzpqv65r2wamkxd9gcvslg6kv-nixos-system-robotaki-21.05pre-git/bin/switch-to-configuration: line 6: use: command not found
[ERROR] /nix/store/2l0hln4hzpqv65r2wamkxd9gcvslg6kv-nixos-system-robotaki-21.05pre-git/bin/switch-to-configuration: line 7: use: command not found
[ERROR] /nix/store/2l0hln4hzpqv65r2wamkxd9gcvslg6kv-nixos-system-robotaki-21.05pre-git/bin/switch-to-configuration: line 8: syntax error near unexpected token `('
[ERROR] /nix/store/2l0hln4hzpqv65r2wamkxd9gcvslg6kv-nixos-system-robotaki-21.05pre-git/bin/switch-to-configuration: line 8: `use Sys::Syslog qw(:standard :macros);'

Introduced somewhere between ee52032 (I used that specific commit... for a while) and c6ac931 . I will try to find time to bisect tomorrow

Minimal repro

shell.nix

{ pkgs ? import <nixpkgs> { } }:
let
  myColmena = import
    (fetchTarball {
      url = "https://github.com/zhaofengli/colmena/archive/c6ac93152cbfe012013e994c5d1108e5008742d6.tar.gz";
      sha256 = "0zljn06yszzy1ghzfd3hyzxwfr9b26iydfgyqwag7h0d8bg2mgjr";
    })
    { };
in
pkgs.mkShell {
  buildInputs = [ myColmena ];
}

hive.nix

let
  nixpkgs = import <nixpkgs>;
in
{
  network = {
    nixpkgs = nixpkgs {
      system = "aarch64-linux";
    };
  };

  alpha = { config, pkgs, ... }: {
    boot.loader.grub.devices = [ "/dev/foo" ];
    fileSystems."/" = {
      device = "/dev/foo";
      fsType = "ext4";
    };

    system.activationScripts.testingOverlays = builtins.throw pkgs.system;
  };
}

Result:

$ colmena build --on alpha
<...snip...>
[ERROR] Evaluation of alpha failed. Logs:
[ERROR] error: x86_64-linux

See also:

• My old PR fixing this initially
• A similar issue in Morph

[justinas]

Updated the repro code, bisecting

[justinas]

So, it seems that your subsequent fix of my PR broke my use-case. d16a136

As in the repro, I use a nixpkgs instantiated with a specific config in my hive. Seems like that config does not get propagated properly.

It looks like doing this instead works properly:

let nixpkgs = ...;
in
{
  network = {
    inherit nixpkgs;
  };
  alpha = {
    nixpkgs.system = "aarch64-linux";
  };
}

I am not sure of a way to fix it for all variants of the very flexible network.nixpkgs argument

[zhaofengli]

I gave this some thought, and I think there are two ways to solve this issue:

1. Add more machinery to merge the configs, similar to what morph previously did but removed (Merge per-machine nixpkgs arguments on top of network-wide ones DBCDK/morph#67, Workaround/fix for issue #68 DBCDK/morph#70).
2. Simplify things and restore the previous behavior in Propagate same 'pkgs' to all modules #12, but setting nixpkgs.pkgs instead of specialArgs.pkgs. This allows nixpkgs.overlays to work from my testing, but it will no longer be possible to set many other nixpkgs.* options on the node level.

For 2, a "compromise" would be that we only set nixpkgs.pkgs if the user pass in an initialized Nixpkgs attribute set in meta.nixpkgs or meta.nodeNixpkgs.

[lovesegfault]

Might be interesting to see how deploy-rs deals with this. I currently use it for my multi-arch network without hitting this kind of problem

[zhaofengli]

I think the usecases mentioned here are now working. The merging of nixpkgs.config is working with 22.11 (currently unstable) thanks to most configs keys now being typed. We have had a warning for this that is now disabled for 22.11+.