R4.0: Qubes won't start anymore: "ERROR: Got empty response from qubesd."

[john-999]

Some Qubes suddenly won't start anymore, the error message being:

"ERROR: Got empty response from qubesd. See journalctl in dom0 for details."

$ journalctl for the exact time span returns:

Apr 11 12:09:24 dom0 lvm[918]: Monitoring thin pool qubes_dom0-pool00-tpool.
Apr 11 12:09:24 dom0 qubesd[16970]: b' WARNING: Sum of all thin volume sizes (2.40 TiB) exceeds the size of thin pool qubes_dom0/pool00 and the size of whole volume group (2.05 TiB)!\n'
Apr 11 12:09:25 dom0 dmeventd[918]: No longer monitoring thin pool qubes_dom0-pool00-tpool.
Apr 11 12:09:25 dom0 lvm[918]: Monitoring thin pool qubes_dom0-pool00-tpool.
Apr 11 12:09:25 dom0 qubesd[16970]: b' WARNING: Sum of all thin volume sizes (2.41 TiB) exceeds the size of thin pool qubes_dom0/pool00 and the size of whole volume group (2.05 TiB)!\n'
Apr 11 12:09:26 dom0 qubesd[16970]: b' WARNING: Sum of all thin volume sizes (2.42 TiB) exceeds the size of thin pool qubes_dom0/pool00 and the size of whole volume group (2.05 TiB)!\n'
Apr 11 12:09:26 dom0 qubesd[16970]: Start failed: 'qubes.devices.UnknownDevice object' has no attribute 'device_node'
Apr 11 12:09:26 dom0 dmeventd[918]: No longer monitoring thin pool qubes_dom0-pool00-tpool.
Apr 11 12:09:26 dom0 lvm[918]: Monitoring thin pool qubes_dom0-pool00-tpool.
Apr 11 12:09:26 dom0 qubesd[16970]: b' WARNING: Sum of all thin volume sizes (2.42 TiB) exceeds the size of thin pool qubes_dom0/pool00 and the size of whole volume group (2.05 TiB)!\n'
Apr 11 12:09:27 dom0 dmeventd[918]: No longer monitoring thin pool qubes_dom0-pool00-tpool.
Apr 11 12:09:27 dom0 lvm[918]: Monitoring thin pool qubes_dom0-pool00-tpool.
Apr 11 12:09:28 dom0 dmeventd[918]: No longer monitoring thin pool qubes_dom0-pool00-tpool.
Apr 11 12:09:28 dom0 lvm[918]: Monitoring thin pool qubes_dom0-pool00-tpool.
Apr 11 12:09:29 dom0 qubesd[16970]: b' WARNING: Sum of all thin volume sizes (2.42 TiB) exceeds the size of thin pool qubes_dom0/pool00 and the size of whole volume group (2.05 TiB)!\n'
Apr 11 12:09:30 dom0 qmemman.daemon.reqhandler[16977]: EOF
Apr 11 12:09:30 dom0 qubesd[16970]: unhandled exception while calling src=b'dom0' meth=b'admin.vm.Start' dest=b'personal' arg=b'' len(untrusted_payload)=0
Apr 11 12:09:30 dom0 qubesd[16970]: Traceback (most recent call last):
Apr 11 12:09:30 dom0 qubesd[16970]: File "/usr/lib/python3.5/site-packages/qubes/api/init.py", line 262, in respond
Apr 11 12:09:30 dom0 qubesd[16970]: untrusted_payload=untrusted_payload)
Apr 11 12:09:30 dom0 qubesd[16970]: File "/usr/lib64/python3.5/asyncio/futures.py", line 381, in iter
Apr 11 12:09:30 dom0 qubesd[16970]: yield self # This tells Task to wait for completion.
Apr 11 12:09:30 dom0 qubesd[16970]: File "/usr/lib64/python3.5/asyncio/tasks.py", line 310, in _wakeup
Apr 11 12:09:30 dom0 qubesd[16970]: future.result()
Apr 11 12:09:30 dom0 qubesd[16970]: File "/usr/lib64/python3.5/asyncio/futures.py", line 294, in result
Apr 11 12:09:30 dom0 qubesd[16970]: raise self._exception
Apr 11 12:09:30 dom0 qubesd[16970]: File "/usr/lib64/python3.5/asyncio/tasks.py", line 240, in _step
Apr 11 12:09:30 dom0 qubesd[16970]: result = coro.send(None)
Apr 11 12:09:30 dom0 qubesd[16970]: File "/usr/lib/python3.5/site-packages/qubes/api/admin.py", line 773, in vm_start
Apr 11 12:09:30 dom0 qubesd[16970]: yield from self.dest.start()
Apr 11 12:09:30 dom0 qubesd[16970]: File "/usr/lib/python3.5/site-packages/qubes/vm/qubesvm.py", line 896, in start
Apr 11 12:09:30 dom0 qubesd[16970]: self._update_libvirt_domain()
Apr 11 12:09:30 dom0 qubesd[16970]: File "/usr/lib/python3.5/site-packages/qubes/vm/qubesvm.py", line 1875, in _update_libvirt_domain
Apr 11 12:09:30 dom0 qubesd[16970]: domain_config = self.create_config_file()
Apr 11 12:09:30 dom0 qubesd[16970]: File "/usr/lib/python3.5/site-packages/qubes/vm/init.py", line 469, in create_config_file
Apr 11 12:09:30 dom0 qubesd[16970]: ]).render(vm=self)
Apr 11 12:09:30 dom0 qubesd[16970]: File "/usr/lib/python3.5/site-packages/jinja2/environment.py", line 989, in render
Apr 11 12:09:30 dom0 qubesd[16970]: return self.environment.handle_exception(exc_info, True)
Apr 11 12:09:30 dom0 qubesd[16970]: File "/usr/lib/python3.5/site-packages/jinja2/environment.py", line 754, in handle_exception
Apr 11 12:09:30 dom0 qubesd[16970]: reraise(exc_type, exc_value, tb)
Apr 11 12:09:30 dom0 qubesd[16970]: File "/usr/lib/python3.5/site-packages/jinja2/_compat.py", line 37, in reraise
Apr 11 12:09:30 dom0 qubesd[16970]: raise value.with_traceback(tb)
Apr 11 12:09:30 dom0 qubesd[16970]: File "/usr/share/qubes/templates/libvirt/xen.xml", line 93, in top-level template code
Apr 11 12:09:30 dom0 qubesd[16970]: {% block devices %}
Apr 11 12:09:30 dom0 qubesd[16970]: File "/usr/share/qubes/templates/libvirt/xen.xml", line 135, in block "devices"
Apr 11 12:09:30 dom0 qubesd[16970]: {% include 'libvirt/devices/block.xml' %}
Apr 11 12:09:30 dom0 qubesd[16970]: File "/usr/share/qubes/templates/libvirt/devices/block.xml", line 3, in top-level template code
Apr 11 12:09:30 dom0 qubesd[16970]:
Apr 11 12:09:30 dom0 qubesd[16970]: jinja2.exceptions.UndefinedError: 'qubes.devices.UnknownDevice object' has no attribute 'device_node'

EDIT: A restored backup yields the same error.

EDIT 2: I am not "on testing", because:

I have not enabled any of the 3 testing repos (https://www.qubes-os.org/doc/software-update-dom0/#testing-repositories): I have checked the "enabled" booleans in /etc/yum.repos.d/qubes-dom0.repo and they are all set to FALSE except of course for [qubes-dom0-current].

And I only remember doing updates using the GUI (which should not get me "on testing", I suppose).

[pdinoto]

Something similar is happening to my R4.0 install. All updates from *-testing.

@john-999, the easiest way to copy debug info from dom0 for me is to create a text file in dom0 (either opening a terminal with vi, or any other means). Paste the debug text onto it (you can copy and paste within dom0 without any special key presses), save the file and then qvm-copy-to-vm <your-internet-connected-AppVM> <the-file-you-have-just-created> and then use the file contents on the AppVM to post here.

My original post on qubes-users mailing list

[pdinoto]

Something similar is happening to my R4.0 install. All updates from *-testing.

@john-999, the easiest way to copy debug info from dom0 for me is to create a text file in dom0 (either opening a terminal with vi, or any other means). Paste the debug text onto it (you can copy and paste within dom0 without any special key presses), save the file and then qvm-copy-to-vm <your-internet-connected-AppVM> <the-file-you-have-just-created> and then use the file contents on the AppVM to post here.

My original post on qubes-users mailing list

[john-999]

Something similar is happening to my R4.0 install. All updates from *-testing.

@pdinoto Did you require the *-testing repo explicitly? Because I didn't (see my Edit 2 above).

But if you did, then it may be the cause of your troubles and you may follow instructions further down this thread to undo the updates: https://groups.google.com/d/msg/qubes-users/MfsCBe9du6s/iKEPmmzRAQAJ

[john-999]

Something similar is happening to my R4.0 install. All updates from *-testing.

@pdinoto Did you require the *-testing repo explicitly? Because I didn't (see my Edit 2 above).

But if you did, then it may be the cause of your troubles and you may follow instructions further down this thread to undo the updates: https://groups.google.com/d/msg/qubes-users/MfsCBe9du6s/iKEPmmzRAQAJ

[marmarek]

Looks like you the VM you're trying to start (personal) have assigned some block device that do not exists anymore. See qvm-block ls personal and detach such device.

[marmarek]

Looks like you the VM you're trying to start (personal) have assigned some block device that do not exists anymore. See qvm-block ls personal and detach such device.

[john-999]

Thank you @marmarek - you're right: I had been using a DVD for safe data import (USB not being safe). Therefor, I had to do the following manually (since QubesOS does not handle integrated optical drives):

# Attach optical drive to TemplateVM:
$ qvm-device block attach --ro --persistent fedora-26-office dom0:sr0
# Mount the optical drive into /media/cdrom
$ sudo mount /dev/xvdi /media/cdrom

And since I had then simply removed the DVD, the VM wasn't able to access it anymore.

Now, does all this mean that I have to manually detach and unmount the DVD drive every time before shutting down the VM?

[john-999]

Thank you @marmarek - you're right: I had been using a DVD for safe data import (USB not being safe). Therefor, I had to do the following manually (since QubesOS does not handle integrated optical drives):

# Attach optical drive to TemplateVM:
$ qvm-device block attach --ro --persistent fedora-26-office dom0:sr0
# Mount the optical drive into /media/cdrom
$ sudo mount /dev/xvdi /media/cdrom

And since I had then simply removed the DVD, the VM wasn't able to access it anymore.

Now, does all this mean that I have to manually detach and unmount the DVD drive every time before shutting down the VM?

[marmarek]

You can use non-persistent mode instead (start the VM first).

[marmarek]

You can use non-persistent mode instead (start the VM first).

[john-999]

But that would also mean that I would have to attach and mount it manually every time I use the VM, right?

[john-999]

But that would also mean that I would have to attach and mount it manually every time I use the VM, right?

[marmarek]

Yes, you have two options:

• persistent mode: the device is always attached, but you can't start the VM if device is not present
• non-persistent mode: you need to attach the device whenever you want to use it

[marmarek]

Yes, you have two options:

• persistent mode: the device is always attached, but you can't start the VM if device is not present
• non-persistent mode: you need to attach the device whenever you want to use it

[john-999]

Funny: I just detached the optical drive from all VMs (using the dom0 terminal), and now, this device won't appear in "Files" (in "Other Locations") when I attach it using the device widget (where it shows attached).

A USB key, on the other hand, attaches and detaches flawlessly in both "Files" and also the device widget.

What logs should I post?

[john-999]

Funny: I just detached the optical drive from all VMs (using the dom0 terminal), and now, this device won't appear in "Files" (in "Other Locations") when I attach it using the device widget (where it shows attached).

A USB key, on the other hand, attaches and detaches flawlessly in both "Files" and also the device widget.

What logs should I post?

[pdinoto]

@john-999: My install is stock R4.0 with sudo qubes-dom0-updates --enablerepo=qubes*testing

Edit: (removing misplaced comment)

[pdinoto]

@john-999: My install is stock R4.0 with sudo qubes-dom0-updates --enablerepo=qubes*testing

Edit: (removing misplaced comment)

[marmarek]

@pdinoto I think you're talking about #3809

[marmarek]

@pdinoto I think you're talking about #3809