Update causes qubes-mount-dirs.service to not start on Whonix (Debian8) template / Missing libxen dependency

[entr0py]

Qubes OS version (e.g., R3.2):

3.1

Affected TemplateVMs (e.g., fedora-23, if applicable):

Whonix-13 (Debian-8)

---

Actual behavior:

System completes boot and achieves Green VM state. /home/user does not mount. No user processes can be executed.

Steps to reproduce the behavior:

Install latest updates (2016-12-04) to dom0 and Whonix templates. Reboot and launch VMs.

General notes:

kernel log shows:

[    1.547644] systemd[1]: Cannot add dependency job for unit rinetd.service, ignoring: Unit rinetd.service failed to load: No such file or directory.
[    1.547676] systemd[1]: Cannot add dependency job for unit rinetd.service, ignoring: Unit rinetd.service failed to load: No such file or directory.
[    1.547772] systemd[1]: Found ordering cycle on local-fs.target/start
[    1.547779] systemd[1]: Found dependency on qubes-mount-dirs.service/start
[    1.547785] systemd[1]: Found dependency on qubes-sysinit.service/start
[    1.547791] systemd[1]: Found dependency on qubes-db.service/start
[    1.547796] systemd[1]: Found dependency on local-fs.target/start
[    1.547802] systemd[1]: Breaking ordering cycle by deleting job qubes-mount-dirs.service/start
[    1.547809] systemd[1]: Job qubes-mount-dirs.service/start deleted to break ordering cycle starting with local-fs.target/start

---

Related issues:

[andrewdavidwong]

Other reports of Debian problems after recent update:

• https://groups.google.com/d/msgid/qubes-users/38621bd7-47a1-4224-bb2c-0f852aa64c5b%40googlegroups.com

• https://twitter.com/legind/status/805576125837557760

[andrewdavidwong]

Other reports of Debian problems after recent update:

• https://groups.google.com/d/msgid/qubes-users/38621bd7-47a1-4224-bb2c-0f852aa64c5b%40googlegroups.com

• https://twitter.com/legind/status/805576125837557760

[qzio]

I can confirm this.

Qubes 3.1
template: Debian-8

[qzio]

I can confirm this.

Qubes 3.1
template: Debian-8

[agutl]

I have the same issue (Qubes 3.1, Debian-8 & whonix). My debian-testing template without applied update still works.

I noticed that the package qubesdb-vm fails to install during the upgrade because it depends on libxen-4.6 (>= 2001:4.6.1-20) but libxen is version 2001:4.6.0-13+deb8u1.

Could this be the root of the issue?

[agutl]

I have the same issue (Qubes 3.1, Debian-8 & whonix). My debian-testing template without applied update still works.

I noticed that the package qubesdb-vm fails to install during the upgrade because it depends on libxen-4.6 (>= 2001:4.6.1-20) but libxen is version 2001:4.6.0-13+deb8u1.

Could this be the root of the issue?

[andrewdavidwong]

In the mean time, you can use qvm-revert-template-changes if you haven't already restarted the affected templates.

[andrewdavidwong]

In the mean time, you can use qvm-revert-template-changes if you haven't already restarted the affected templates.

[qzio]

@andrewdavidwong Thanks for a quck workaround suggestion,
Sadly, that workaround doesn't work for me...

When I first encountered this issue, I restarted my appvm and templatevm..
I then shut them down and run qvm-revert-template-changes which looked ok.
But I'm not able to run any command in either templateVM or appVMs.

Sadly I don't have a reasonable up to date backup of my templateVM either.

Not sure if this is the appropriate place to ask for workarounds, but is it possible somehow to get /home/user from an appVM mounted somewhere so I can extract some files?

[qzio]

@andrewdavidwong Thanks for a quck workaround suggestion,
Sadly, that workaround doesn't work for me...

When I first encountered this issue, I restarted my appvm and templatevm..
I then shut them down and run qvm-revert-template-changes which looked ok.
But I'm not able to run any command in either templateVM or appVMs.

Sadly I don't have a reasonable up to date backup of my templateVM either.

Not sure if this is the appropriate place to ask for workarounds, but is it possible somehow to get /home/user from an appVM mounted somewhere so I can extract some files?

[agutl]

@qzio You could try to reinstall the debian-8 template (https://www.qubes-os.org/doc/reinstall-template/) but I don't know if it already contains the latest updates.
I'll try that (to late for qvm-revert-template-changes for me, too).

[agutl]

@qzio You could try to reinstall the debian-8 template (https://www.qubes-os.org/doc/reinstall-template/) but I don't know if it already contains the latest updates.
I'll try that (to late for qvm-revert-template-changes for me, too).

[marmarek]

Missing libxen-4.6 update seems to be the cause, just uploaded the update. Also, enabling testing repository should be enough (the package was there, but I missed uploading it to stable).

So, to fix the issue - start the template, access its console (sudo xl console debian-8) and install updates (sudo apt-get update && sudo apt-get -V dist-upgrade).

[marmarek]

Missing libxen-4.6 update seems to be the cause, just uploaded the update. Also, enabling testing repository should be enough (the package was there, but I missed uploading it to stable).

So, to fix the issue - start the template, access its console (sudo xl console debian-8) and install updates (sudo apt-get update && sudo apt-get -V dist-upgrade).

[andrewdavidwong]

@qzio:

Not sure if this is the appropriate place to ask for workarounds, but is it possible somehow to get /home/user from an appVM mounted somewhere so I can extract some files?

Your files are stored in /var/lib/qubes/appvms/<vm-name>/private.img. You can (re)install a fresh template from the repos as @agutl suggested, then base the AppVM on the fresh template to start it. Or you can try manually mounting private.img if that doesn't work, but I don't recommend doing this in dom0.

[andrewdavidwong]

@qzio:

Not sure if this is the appropriate place to ask for workarounds, but is it possible somehow to get /home/user from an appVM mounted somewhere so I can extract some files?

Your files are stored in /var/lib/qubes/appvms/<vm-name>/private.img. You can (re)install a fresh template from the repos as @agutl suggested, then base the AppVM on the fresh template to start it. Or you can try manually mounting private.img if that doesn't work, but I don't recommend doing this in dom0.

[agutl]

@marmarek Thx, just did that and it solves the issue :).

edit: Now I get ERROR: Failed to connect to qmemman: [Errno 111] Connection refused but that's probably unrelated. Hopefully I'm not offline after a restart...
edit2: Unrelated and gone after reboot.

[agutl]

@marmarek Thx, just did that and it solves the issue :).

edit: Now I get ERROR: Failed to connect to qmemman: [Errno 111] Connection refused but that's probably unrelated. Hopefully I'm not offline after a restart...
edit2: Unrelated and gone after reboot.

[qzio]

@marmarek thanks, this solved it for me as well.

@andrewdavidwong thank you for explaining!

For ref: sudo x1 console debian-8 did not work for me.

x1 command not found

But I was able to access the console using virsh

# virsh
virsh #  connect xen
virsh #  console debian-8

[qzio]

@marmarek thanks, this solved it for me as well.

@andrewdavidwong thank you for explaining!

For ref: sudo x1 console debian-8 did not work for me.

x1 command not found

But I was able to access the console using virsh

# virsh
virsh #  connect xen
virsh #  console debian-8

[adrelanos]

@qzio: xl not x1. (Character l. Not digit 1.)

[adrelanos]

@qzio: xl not x1. (Character l. Not digit 1.)

[entr0py]

Thanks all!

qvm-revert-template-changes is a lifesaver.

is it possible somehow to get /home/user from an appVM mounted

FWIW, you could also have manually started the mount-dirs service in the broken appVM:
sudo systemctl start qubes-mount-dirs.service

[entr0py]

Thanks all!

qvm-revert-template-changes is a lifesaver.

is it possible somehow to get /home/user from an appVM mounted

FWIW, you could also have manually started the mount-dirs service in the broken appVM:
sudo systemctl start qubes-mount-dirs.service

[adrelanos]

3n7r0p1:

`qvm-revert-template-changes` is a lifesaver.

related: https://forums.whonix.org/t/qubes-vm-snapshots-using-git-svn

[adrelanos]

3n7r0p1:

`qvm-revert-template-changes` is a lifesaver.

related: https://forums.whonix.org/t/qubes-vm-snapshots-using-git-svn