Virtualbox deploy failed because of VirtualBox Guest Services failure

[mickours]

Hi,

I'm trying the NixOps tutorial and I'm stuck with this error: Exception: unable to activate new configuration
It seems to be a problem with the Guest Service of virtualbox that fails to connect to some property server. Here is the logs from journalctl inside the VM:

Mar 19 08:57:34 webserver systemd[1]: Started VirtualBox Guest Services.
Mar 19 08:57:34 webserver VBoxService[1808]: VBoxService 5.2.6 r120293 (verbosity: 0) linux.amd64 (Jan 15 2018 14:51:00) release log
Mar 19 08:57:34 webserver VBoxService[1808]: 00:00:00.000605 main     Log opened 2018-03-19T08:57:34.270255000Z
Mar 19 08:57:34 webserver VBoxService[1808]: 00:00:00.000673 main     OS Product: Linux
Mar 19 08:57:34 webserver VBoxService[1808]: 00:00:00.000699 main     OS Release: 4.4.24
Mar 19 08:57:34 webserver VBoxService[1808]: 00:00:00.000717 main     OS Version: #1-NixOS SMP Fri Oct 7 13:23:59 UTC 2016
Mar 19 08:57:34 webserver VBoxService[1808]: 00:00:00.000735 main     Executable: /nix/store/8gcingpvcfw1084xcs490b48p1w6wyj4-VirtualBox-GuestAdditions-5.2.6-4.14.23/bin/VBoxService
Mar 19 08:57:34 webserver VBoxService[1808]: 00:00:00.000736 main     Process ID: 1808
Mar 19 08:57:34 webserver VBoxService[1808]: 00:00:00.000737 main     Package type: LINUX_64BITS_GENERIC
Mar 19 08:57:34 webserver VBoxService[1808]: 00:00:00.002071 main     Error: Failed to connect to the guest property service, rc=VERR_INTERNAL_ERROR
Mar 19 08:57:34 webserver VBoxService[1808]: 00:00:00.002112 main     Error: Service 'control' failed pre-init: VERR_INTERNAL_ERROR
Mar 19 08:57:34 webserver systemd[1]: virtualbox.service: Main process exited, code=exited, status=1/FAILURE
Mar 19 08:57:34 webserver systemd[1]: virtualbox.service: Failed with result 'exit-code'.

I'm using the very last NixOps version built from source because I already encounter an other bug that was solved upstream (#875).

I'm using an unstable NixOS version:

$ nixos-version
18.03pre130569.7a04c2ca296 (Impala)

Here is the deploy execution log:

$ nixops deploy -d trivial
building all machine configurations...
webserver> copying closure...
trivial> closures copied successfully
webserver> updating GRUB 2 menu...
webserver> activating the configuration...
webserver> setting up /etc...
webserver> setting up tmpfiles
webserver> warning: the following units failed: get-vbox-nixops-client-key.service, virtualbox.service
webserver> 
webserver> ● virtualbox.service - VirtualBox Guest Services
webserver>    Loaded: loaded (/nix/store/6f6ryks4ds2d2hrqrs60a5x3j5c8vs51-unit-virtualbox.service/virtualbox.service; enabled; vendor preset: enabled)
webserver>    Active: failed (Result: exit-code) since Mon 2018-03-19 09:22:27 UTC; 71ms ago
webserver>   Process: 2229 ExecStart=VBoxService --foreground (code=exited, status=1/FAILURE)
webserver>  Main PID: 2229 (code=exited, status=1/FAILURE)
webserver> 
webserver> Mar 19 09:22:27 webserver systemd[1]: Started VirtualBox Guest Services.
webserver> Mar 19 09:22:27 webserver systemd[1]: virtualbox.service: Main process exited, code=exited, status=1/FAILURE
webserver> Mar 19 09:22:27 webserver systemd[1]: virtualbox.service: Failed with result 'exit-code'.
webserver> 
webserver> ● get-vbox-nixops-client-key.service - Get NixOps SSH Key
webserver>    Loaded: loaded (/nix/store/ajj6bb9nq20w6c5qq9k1npvz9vc1qnv6-unit-get-vbox-nixops-client-key.service/get-vbox-nixops-client-key.service; enabled; vendor preset: enabled)
webserver>    Active: failed (Result: exit-code) since Mon 2018-03-19 09:22:27 UTC; 66ms ago
webserver>   Process: 2226 ExecStart=/nix/store/1y5g8yw8xj2d77xrg5a7hxgry9jgvlfg-unit-script/bin/get-vbox-nixops-client-key-start (code=exited, status=1/FAILURE)
webserver>  Main PID: 2226 (code=exited, status=1/FAILURE)
webserver> 
webserver> Mar 19 09:22:27 webserver systemd[1]: Started Get NixOps SSH Key.
webserver> Mar 19 09:22:27 webserver get-vbox-nixops-client-key-start[2226]: VBoxControl: error: Failed to connect to the guest property service, error VERR_INTERNAL_ERROR
webserver> Mar 19 09:22:27 webserver systemd[1]: get-vbox-nixops-client-key.service: Main process exited, code=exited, status=1/FAILURE
webserver> Mar 19 09:22:27 webserver systemd[1]: get-vbox-nixops-client-key.service: Failed with result 'exit-code'.
webserver> error: Traceback (most recent call last):
  File "/nix/store/6zyvpi0q6mvprycadz2dpdqag4742y18-python2.7-nixops-1.6pre0_abcdef/lib/python2.7/site-packages/nixops/deployment.py", line 731, in worker
    raise Exception("unable to activate new configuration")
Exception: unable to activate new configuration

error: activation of 1 of 1 machines failed (namely on ‘webserver’)

Note that I'm in unstable so I not complaining that it is not working out of the box ;)
But even if I'm very enthusiastic to discover NixOps, two bug before the end of the tutorial is a bit hard to take. Is there some kind of CI for NixOps? It would be interesting to test the entire tutorial on the unstable version and even add a badge on the tutorial page would be great!

[ElvishJerricco]

My gut feeling is that this has something to do with the fact that the base image starts at NixOS 16.09, which is pretty old, then gets updated to the system you're deploying. I guess 16.09 can't activate an 18.03 config. As a workaround, you can use nixops deploy --force-reboot to force it to merely boot the config and reboot into it.

[andrewchambers]

Confirming this issue affects me too,

 - system: `"x86_64-linux"`
 - host os: `Linux 4.14.32, NixOS, 18.03.131807.489a14add9a (Impala)`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.0`
 - channels(root): `"nixos-18.03.131807.489a14add9a"`
 - channels(ac): `""`
 - nixpkgs: `/home/ac/src/nixpkgs`

[rbrewer123]

I hit this too, running an up-to-date nixos 18.03. --force-reboot seemed to work around it.

[rbrewer123]

@ElvishJerricco what is needed to get the base image modernized? I'm new to NixOS and the path forward for fixing this isn't at all clear to me.

[rbrewer123]

I found more details on rebuilding the base image in #835.

[rbrewer123]

Looks like @rbvermaa fixed this in nixops master by making the virtualbox base image use 18.03. So it's just not in a release version yet. I confirmed I can deploy the webserver from the tutorial using virtualbox without the workaround.

[dugwood]

Hit a similar issue, not on NixOS, but for my Debian guest (development server) inside a Debian host (desktop computer), so that may be linked if your OS was already installed and running. If you installed it from scratch, sorry it won't help I guess.

I was trying to install virtualbox-guest-dkms and virtualbox-guest-utils, from Debian Stretch backports repository (VirtualBox 5.2.10).

Journal log:

mai 08 04:31:10 debian-vm virtualbox-guest-utils[1904]: 00:00:00.005413 main     OS Product: Linux
mai 08 04:31:10 debian-vm virtualbox-guest-utils[1904]: 00:00:00.005787 main     OS Release: 4.9.0-6-amd64
mai 08 04:31:10 debian-vm kernel: VGDrvCommonIoCtl: Unknown request iFunction=0x5609 stripped size=0xc0005609
mai 08 04:31:10 debian-vm kernel: VGDrvCommonIoCtl: Unknown request iFunction=0x5609 stripped size=0xc0005609
mai 08 04:31:10 debian-vm kernel: VGDrvCommonIoCtl: Unknown request iFunction=0xc09c5604 stripped size=0xc0005604
mai 08 04:31:10 debian-vm kernel: VGDrvCommonIoCtl: Unknown request iFunction=0x5609 stripped size=0xc0005609
mai 08 04:31:10 debian-vm kernel: VGDrvCommonIoCtl: Unknown request iFunction=0x5609 stripped size=0xc0005609
mai 08 04:31:10 debian-vm virtualbox-guest-utils[1904]: 00:00:00.011760 main     OS Version: #1 SMP Debian 4.9.88-1 (2018-04-29)
mai 08 04:31:10 debian-vm virtualbox-guest-utils[1904]: 00:00:00.012053 main     Executable: /usr/sbin/VBoxService
mai 08 04:31:10 debian-vm virtualbox-guest-utils[1904]: 00:00:00.012056 main     Process ID: 1937
mai 08 04:31:10 debian-vm virtualbox-guest-utils[1904]: 00:00:00.012059 main     Package type: LINUX_64BITS_GENERIC (OSE)
mai 08 04:31:10 debian-vm virtualbox-guest-utils[1904]: 00:00:00.013547 main     Error: Failed to connect to the guest property service, rc=VERR_INTERNAL_ERROR
mai 08 04:31:10 debian-vm virtualbox-guest-utils[1904]: 00:00:00.013749 main     Error: Service 'control' failed pre-init: VERR_INTERNAL_ERROR
mai 08 04:31:10 debian-vm virtualbox-guest-utils[1904]:  failed!
mai 08 04:31:10 debian-vm systemd[1]: virtualbox-guest-utils.service: Control process exited, code=exited status=1
mai 08 04:31:10 debian-vm systemd[1]: Failed to start Virtualbox guest utils.
-- Subject: L'unité (unit) virtualbox-guest-utils.service a échoué

After removing virtualbox-guest-dkms, and restarting the VM, I can see that an old Virtualbox is still active, using dmesg|grep vbox:

kernel: vboxguest: Successfully loaded version 5.1.30

That's the one I've installed using VirtualBox guest additions' ISO, which was a real pain on kernel upgrades, so I've switched to DKMS.

Okay, found the culprit with find /lib/ -name "vbox*":

/lib/modules/4.9.0-6-amd64/misc/vboxguest.ko
/lib/modules/4.9.0-6-amd64/misc/vboxsf.ko
/lib/modules/4.9.0-6-amd64/misc/vboxvideo.ko

Removing the files... running depmod -a... reboot...

Got the expected Failed to find module 'vboxsf' as I forgot to remove it from /etc/modules.

Reinstalling: virtualbox-guest-dkms and virtualbox-guest-utils... reboot...

debian-vm# systemctl status virtualbox-guest-utils.service 
 virtualbox-guest-utils.service - Virtualbox guest utils
   Loaded: loaded (/lib/systemd/system/virtualbox-guest-utils.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-05-08 05:44:04 CEST; 1min 38s ago
  Process: 311 ExecStart=/etc/init.d/virtualbox-guest-utils start (code=exited, status=0/SUCCESS)

Hooray!

[turboMaCk]

I've run into this as well.

❯❯❯ cat /etc/os-release 
NAME=NixOS
ID=nixos
VERSION="18.09.1974.50f41ea2fcf (Jellyfish)"
VERSION_CODENAME=jellyfish
VERSION_ID="18.09.1974.50f41ea2fcf"
PRETTY_NAME="NixOS 18.09.1974.50f41ea2fcf (Jellyfish)"
HOME_URL="https://nixos.org/"
SUPPORT_URL="https://nixos.org/nixos/support.html"
BUG_REPORT_URL="https://github.com/NixOS/nixpkgs/issues"