nixops deploy puts server in unusable state

[pikajude]

jude-web> copying closure...
jude-web> updating GRUB 2 menu...
jude-web> installing the GRUB 2 boot loader on /dev/sda...
jude-web> Installing for i386-pc platform.
jude-web> You have a memory leak (not released memory pool):
jude-web>  [0x1c3ae10] dtree
jude-web> Internal error: Unreleased memory pool(s) found.
jude-web> You have a memory leak (not released memory pool):
jude-web>  [0x1c3ae10] dtree
jude-web> Internal error: Unreleased memory pool(s) found.
jude-web> You have a memory leak (not released memory pool):
jude-web>  [0x1c3ae10] dtree
jude-web> Internal error: Unreleased memory pool(s) found.
jude-web> You have a memory leak (not released memory pool):
jude-web>  [0x1c3ae10] dtree
jude-web> Internal error: Unreleased memory pool(s) found.
jude-web> You have a memory leak (not released memory pool):
jude-web>  [0x1c3ae10] dtree
jude-web> Internal error: Unreleased memory pool(s) found.
jude-web> You have a memory leak (not released memory pool):
jude-web>  [0x1c3ae10] dtree
jude-web> Internal error: Unreleased memory pool(s) found.
jude-web> You have a memory leak (not released memory pool):
jude-web>  [0x1c3ae10] dtree
jude-web> Internal error: Unreleased memory pool(s) found.
jude-web> You have a memory leak (not released memory pool):
jude-web>  [0x1c3ae10] dtree
jude-web> Internal error: Unreleased memory pool(s) found.
jude-web> You have a memory leak (not released memory pool):
jude-web>  [0x1c38e20] dtree
jude-web> Internal error: Unreleased memory pool(s) found.
jude-web> You have a memory leak (not released memory pool):
jude-web>  [0x1c38e20] dtree
jude-web> Internal error: Unreleased memory pool(s) found.
jude-web> You have a memory leak (not released memory pool):
jude-web>  [0x1c3c320] dtree
jude-web> Internal error: Unreleased memory pool(s) found.
jude-web> You have a memory leak (not released memory pool):
jude-web>  [0x1c3c320] dtree
jude-web> Internal error: Unreleased memory pool(s) found.
jude-web> Installation finished. No error reported.
jude-web> stopping swap device: /dev/disk/by-label/swap
jude-web> stopping the following units: -.mount, alsa-store.service, cpufreq.service, kmod-static-nodes.service, network-local-commands.service, network-setup.service, nix-daemon.service, nscd.service, ntpd.service, systemd-logind.service, systemd-modules-load.service, systemd-sysctl.service, systemd-tmpfiles-clean.timer, systemd-tmpfiles-setup-dev.service, systemd-udev-settle.service, systemd-udev-trigger.service, systemd-udevd.service, systemd-vconsole-setup.service
jude-web> Failed to stop -.mount: Operation refused, unit -.mount may be requested by dependency only.
jude-web> Warning: Stopping nix-daemon.service, but it can still be activated by:
jude-web>   nix-daemon.socket
jude-web> Warning: Stopping systemd-udevd.service, but it can still be activated by:
jude-web>   systemd-udevd-control.socket
jude-web>   systemd-udevd-kernel.socket
jude-web> NOT restarting the following changed units: getty@tty1.service, systemd-backlight@backlight:acpi_video0.service, systemd-journal-flush.service, systemd-journald.service, systemd-random-seed.service, systemd-remount-fs.service, systemd-tmpfiles-setup.service, systemd-update-utmp.service, systemd-user-sessions.service, user@0.service
jude-web> activating the configuration...
jude-web> setting up /etc...
jude-web> regenerating udev hardware database...
jude-web> restarting systemd...
jude-web> reloading the following units: dbus.service, dev-hugepages.mount, dev-mqueue.mount, firewall.service, sys-kernel-debug.mount
jude-web> restarting the following units: dhcpcd.service, sshd.service
jude-web> starting the following units: -.mount, alsa-store.service, cpufreq.service, kmod-static-nodes.service, network-local-commands.service, network-setup.service, nix-daemon.service, nscd.service, ntpd.service, systemd-logind.service, systemd-modules-load.service, systemd-sysctl.service, systemd-tmpfiles-clean.timer, systemd-tmpfiles-setup-dev.service, systemd-udev-settle.service, systemd-udev-trigger.service, systemd-udevd.service, systemd-vconsole-setup.service
jude-web> Failed to start -.mount: Operation refused, unit -.mount may be requested by dependency only.

Then the SSH connection times out because the box has been rebooted into recovery mode, and is no longer accessible.

If nixops deploy is planning to brick my box, I would at least like the option to rollback before it tries to finalize the deploy.

[pikajude]

hardware-configuration.nix:

# Do not modify this file!  It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations.  Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, ... }:

{
  imports = [ ];

  boot.initrd.availableKernelModules = [ ];
  boot.initrd.kernelModules = [ "md_mod" "mpt2sas" "ahci" "raid1" ];
  boot.initrd.mdadmConf = ''
    ARRAY /dev/md0 metadata=1.2 name=rescue.ovh.net:0 UUID=b752989f:36ea045c:5c52d868:c4186964
  '';
  boot.kernelModules = [ "kvm-intel" ];
  boot.extraModulePackages = [ ];

  fileSystems."/" =
    { label = "system";
      fsType = "ext4";
    };

  swapDevices =
    [ { label = "swap"; }
    ];

  nix.maxJobs = 8;
}

[aszlig]

What is your network expression and what's the version of <nixpkgs> you're using?

[edolstra]

When you say "recovery mode", do you mean the systemd unit emergency.service? That would be the consequence of -.mount failing.

The EC2 and GCE NixOS modules import profiles/headless.nix, which sets systemd.enableEmergencyMode = false; to prevent this kind of situation. Maybe NixOps should set that for all deployments, since they are generally remote/headless.

[pikajude]

@aszlig My network expression is spread over 12 or so files, should I manually merge them all or is there any easier/more automated way to do it?

My nixpkgs is on dc39c2dd20f87b65eebb6d3429155cbb71f4a56d. Originally, I ran nixos-install from the nixos-unstable channel on this server, and now I'm trying to nixops deploy to it.

[pikajude]

@edolstra Thank you for the tip. Without falling into emergency mode, my server is at least able to be debugged.

[pikajude]

@aszlig, @edolstra: The problem turned out to be my own mistake: I was pulling in a configuration file I used for a Linode, which has a different disk configuration.