qubesd crashed during boot, no VM started

[ctrlaltdel]

Qubes OS version (e.g., R3.2):

R4.0-rc1

Hardware

Thinkpad X1 Carbon 5th (20HR002RMZ)

---

Expected behavior:

When starting the system, all system VMs (sys-net, sys-firewall and sys-usb) should automatically start during the boot process.

Actual behavior:

No VM is automatically started because qubesd crashes during the boot process, so the system cannot get any network connectivity.

Steps to reproduce the behavior:

This happens at each boot, except the first one after installation.

General notes:

After a fresh start, the following systemd services are in failed state:

• qubesd.service
• qubes-vm@sys-firewall.service
• qubes-vm@sys-net.service
• qubes-vm@sys-usb.service

The root cause seems to be qubesd crashing with the following (double!) exception:

Aug 05 14:31:58 dom0 qubesd[2048]: Traceback (most recent call last):
Aug 05 14:31:58 dom0 qubesd[2048]:   File "/usr/lib/python3.5/site-packages/qubes/storage/__init__.py", line 690, in __getitem__
Aug 05 14:31:58 dom0 qubesd[2048]:     return self._pool.get_volume(item)
Aug 05 14:31:58 dom0 qubesd[2048]:   File "/usr/lib/python3.5/site-packages/qubes/storage/__init__.py", line 794, in get_volume
Aug 05 14:31:58 dom0 qubesd[2048]:     raise self._not_implemented("get_volume")
Aug 05 14:31:58 dom0 qubesd[2048]: NotImplementedError: Pool driver ThinPool has get_volume() not implemented
Aug 05 14:31:58 dom0 qubesd[2048]: During handling of the above exception, another exception occurred:
Aug 05 14:31:58 dom0 qubesd[2048]: Traceback (most recent call last):
Aug 05 14:31:58 dom0 qubesd[2048]:   File "/usr/bin/qubesd", line 9, in <module>
Aug 05 14:31:58 dom0 qubesd[2048]:     load_entry_point('qubes==4.0.4', 'console_scripts', 'qubesd')()
Aug 05 14:31:58 dom0 qubesd[2048]:   File "/usr/lib/python3.5/site-packages/qubes/tools/qubesd.py", line 33, in main
Aug 05 14:31:58 dom0 qubesd[2048]:     args = parser.parse_args(args)
Aug 05 14:31:58 dom0 qubesd[2048]:   File "/usr/lib/python3.5/site-packages/qubes/tools/__init__.py", line 363, in parse_args
Aug 05 14:31:58 dom0 qubesd[2048]:     offline_mode=namespace.offline_mode)
Aug 05 14:31:58 dom0 qubesd[2048]:   File "/usr/lib/python3.5/site-packages/qubes/app.py", line 772, in __init__
Aug 05 14:31:58 dom0 qubesd[2048]:     self.load(lock=lock)
Aug 05 14:31:58 dom0 qubesd[2048]:   File "/usr/lib/python3.5/site-packages/qubes/app.py", line 836, in load
Aug 05 14:31:58 dom0 qubesd[2048]:     vm.fire_event('domain-load')
Aug 05 14:31:58 dom0 qubesd[2048]:   File "/usr/lib/python3.5/site-packages/qubes/events.py", line 195, in fire_event
Aug 05 14:31:58 dom0 qubesd[2048]:     pre_event=pre_event)
Aug 05 14:31:58 dom0 qubesd[2048]:   File "/usr/lib/python3.5/site-packages/qubes/events.py", line 163, in _fire_event
Aug 05 14:31:58 dom0 qubesd[2048]:     effect = func(self, event, **kwargs)
Aug 05 14:31:58 dom0 qubesd[2048]:   File "/usr/lib/python3.5/site-packages/qubes/vm/qubesvm.py", line 739, in on_domain_init_loaded
Aug 05 14:31:58 dom0 qubesd[2048]:     self.storage = qubes.storage.Storage(self)
Aug 05 14:31:58 dom0 qubesd[2048]:   File "/usr/lib/python3.5/site-packages/qubes/storage/__init__.py", line 352, in __init__
Aug 05 14:31:58 dom0 qubesd[2048]:     self.init_volume(name, conf)
Aug 05 14:31:58 dom0 qubesd[2048]:   File "/usr/lib/python3.5/site-packages/qubes/storage/__init__.py", line 388, in init_volume
Aug 05 14:31:58 dom0 qubesd[2048]:     volume_config['source'] = pool.volumes[volume_config['source']]
Aug 05 14:31:58 dom0 qubesd[2048]:   File "/usr/lib/python3.5/site-packages/qubes/storage/__init__.py", line 697, in __getitem__
Aug 05 14:31:58 dom0 qubesd[2048]:     raise KeyError(item)
Aug 05 14:31:58 dom0 qubesd[2048]: KeyError: 'qubes_dom0/vm-fedora-25-root'
Aug 05 14:31:59 dom0 libvirtd[2087]: 2017-08-05 18:31:59.004+0000: 2087: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error
Aug 05 14:31:59 dom0 systemd[1]: qubesd.service: Main process exited, code=exited, status=1/FAILURE
Aug 05 14:31:59 dom0 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubesd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Aug 05 14:31:59 dom0 systemd[1]: Failed to start Qubes OS daemon.
-- Subject: Unit qubesd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit qubesd.service has failed.
-- 
-- The result is failed.
Aug 05 14:31:59 dom0 systemd[1]: qubesd.service: Unit entered failed state.
Aug 05 14:31:59 dom0 systemd[1]: qubesd.service: Failed with result 'exit-code'.

As a workaround, restarting all four services manually, logout of the xfce4 session and log back in seems to work pretty reliably:

sudo systemctl start qubesd
sudo systemctl start qubes-vm@sys-firewall.service
sudo systemctl start qubes-vm@sys-net.service
sudo systemctl start qubes-vm@sys-usb.service
xfce4-session-logout

[0spinboson]

Question: did you update dom0 fc25 at all before first reboot (if you didn't, my related bug report's title is inaccurate :) )?

[0spinboson]

Question: did you update dom0 fc25 at all before first reboot (if you didn't, my related bug report's title is inaccurate :) )?

[ctrlaltdel]

@0spinboson yes, good point, I've probably done it before rebooting.

Do you see the same exception in journalctl? If yes, then this issue is indeed a duplicate of yours (#2960)

[ctrlaltdel]

@0spinboson yes, good point, I've probably done it before rebooting.

Do you see the same exception in journalctl? If yes, then this issue is indeed a duplicate of yours (#2960)

[0spinboson]

I saw the same error after Marek prodded me to actually look at the log (I'm hopeless), yes.

[0spinboson]

I saw the same error after Marek prodded me to actually look at the log (I'm hopeless), yes.

[ctrlaltdel]

Closing as a duplicate of #2960.

[ctrlaltdel]

Closing as a duplicate of #2960.

[0spinboson]

are your problems fixed by the updates currently available in current-testing?
(Update: it seems I spoke too soon. Rebooted twice, and now it's back to being broken again.)

[0spinboson]

are your problems fixed by the updates currently available in current-testing?
(Update: it seems I spoke too soon. Rebooted twice, and now it's back to being broken again.)