Unable to launch applications unless through xl console, networking broken, vchan module warnings

[esote]

Qubes OS version:

R4

Affected component(s):

Specific StandaloneVM

---

Steps to reproduce the behavior:

Expected behavior:

Everything works as should be expected when using Qubes.

Actual behavior:

I run a standalone VM (from Fedora 26 template), which I use for my coding environment. Out of the blue, I was unable to launch applications from the app menu. I tried using qvm-run code xterm, but this did nothing. In order to launch anything, I have to do (in dom0):

sudo xl console code

*login as root*

su user
gnome-terminal

Once I have this terminal up, I can then run journalctl -r, which shows the following red errors:

pulseaudio[1096]: Failed to initialize daemon.
pulseaudio[1096]: Module load failed.
pulseaudio[1096]: Failed to load module "module-vchan-sink" (argument: ""): initialization failed.
pulseaudio[1096]: get_early_allocated_vchan: Permission denied
pulseaudio[1096]: libvchan_server_init play failed
pulseaudio[1096]: vchan module loading

Using qvm-copy from within the VM throws an error:

Data vchan connection failed

This error can be avoided by using sudo, even though the files i am copying are not owned by root (&etc)

As well, the VM has no networking, even though it is using sys-firewall as a netvm, and networking for all other VMs is working fine (including other standalone VMs).

The VM also randomly halts after a while, kicking me out of xl console. I also notice that, when viewing from xl console, in the poweroff messages it shows

[FAILED] Failed unmounting /usr/lib/modules/4.14.18-1.pvops.qubes.x86_64

As well, I am unable to use qvm-sync-appmenus:

Traceback (most recent call last):
  File "/usr/bin/qvm-sync-appmenus", line 9, in <module>
    load_entry_point('qubesdesktop==4.0.12', 'console_scripts', 'qvm-sync-appmenus')()
  File "/usr/lib/python3.5/site-packages/qubesappmenus/receive.py", line 390, in main
    new_appmenus = retrieve_appmenus_templates(vm, use_stdin=use_stdin)
  File "/usr/lib/python3.5/site-packages/qubesappmenus/receive.py", line 364, in retrieve_appmenus_templates
    new_appmenus = get_appmenus(vm if not use_stdin else None)
  File "/usr/lib/python3.5/site-packages/qubesappmenus/receive.py", line 158, in get_appmenus
    "Error getting application list")
qubesadmin.exc.QubesException: Error getting application list

I've also noticed that the memory of the VM (code) is not expanding beyond the initial memory, it just caps out at 399 MB (I have initial set to 400 MB & max to 4000 MB).

General notes:

I do not know what logs to look at to diagnose this problem, so guidance there would be appreciated. I am not sure if the things I have found are part of one bug, or multiple.

I am currently taring directories and qvm-copying them to other VMs in case other things decide to break.

---

Related issues:

[marmarek]

Looks like for some reason vchan connections failed preventing start of:

• pulseaudio (you've found that already)
• qrexec (responsible for qvm-run and qvm-sync-appmenus)
• qubesdb (responsible for network configuration)

Check permissions of /dev/xen/* files - should have rw for group qubes. Have you installed some updates recently? Or changed some configuration?

[marmarek]

Looks like for some reason vchan connections failed preventing start of:

• pulseaudio (you've found that already)
• qrexec (responsible for qvm-run and qvm-sync-appmenus)
• qubesdb (responsible for network configuration)

Check permissions of /dev/xen/* files - should have rw for group qubes. Have you installed some updates recently? Or changed some configuration?

[esote]

@marmarek Using ls -l /dev/xen shows the following (notably no group perms):

crw------- 1 root root 10, 57 May  6 16:57 evtchn
crw------- 1 root root 10, 55 May  6 16:57 gntalloc
crw------- 1 root root 10, 54 May  6 16:57 gntdev
crw------- 1 root root 10, 56 May  6 16:57 privcmd
crw------- 1 root root 10, 62 May  6 16:57 xenbus

What could have changed the permissions on those files?

No, I haven't updated anything crazy really, just a few odd packages from using dsync. I haven't changed any configuration outside of switching the NetVM between sys-firewall and none.

What perms should the files under /dev/xen/* have (and should I change them to fix this?)

[esote]

@marmarek Using ls -l /dev/xen shows the following (notably no group perms):

crw------- 1 root root 10, 57 May  6 16:57 evtchn
crw------- 1 root root 10, 55 May  6 16:57 gntalloc
crw------- 1 root root 10, 54 May  6 16:57 gntdev
crw------- 1 root root 10, 56 May  6 16:57 privcmd
crw------- 1 root root 10, 62 May  6 16:57 xenbus

What could have changed the permissions on those files?

No, I haven't updated anything crazy really, just a few odd packages from using dsync. I haven't changed any configuration outside of switching the NetVM between sys-firewall and none.

What perms should the files under /dev/xen/* have (and should I change them to fix this?)

[marmarek]

Should be this:

crw-rw---- 1 root qubes 10, 56 Apr 22 22:37 evtchn
crw-rw---- 1 root qubes 10, 55 Apr 22 22:37 gntalloc
crw-rw---- 1 root qubes 10, 54 Apr 22 22:37 gntdev
crw-rw---- 1 root qubes 10, 57 Apr 22 22:37 privcmd
crw-rw---- 1 root qubes 10, 62 Apr 22 22:37 xenbus

There are udev rules at /usr/lib/udev/rules.d/99-qubes-misc.rules to enforce this. Do you have this file? Or maybe some other rules later overriding it? Or some udev problem (see logs)?

[marmarek]

Should be this:

crw-rw---- 1 root qubes 10, 56 Apr 22 22:37 evtchn
crw-rw---- 1 root qubes 10, 55 Apr 22 22:37 gntalloc
crw-rw---- 1 root qubes 10, 54 Apr 22 22:37 gntdev
crw-rw---- 1 root qubes 10, 57 Apr 22 22:37 privcmd
crw-rw---- 1 root qubes 10, 62 Apr 22 22:37 xenbus

There are udev rules at /usr/lib/udev/rules.d/99-qubes-misc.rules to enforce this. Do you have this file? Or maybe some other rules later overriding it? Or some udev problem (see logs)?

[esote]

@marmarek My code VM does not have the /usr/lib/udev/rules.d/99-qubes-misc.rules file or any other 99-qubes-* files, but the rest of my VMs do have those files...

Once I changed the permissions on /dev/xen/* (like you suggested) and rebooted the VM the permissions were back to as they were before (no group perms, etc.).

What logs should I look at for udev?

[esote]

@marmarek My code VM does not have the /usr/lib/udev/rules.d/99-qubes-misc.rules file or any other 99-qubes-* files, but the rest of my VMs do have those files...

Once I changed the permissions on /dev/xen/* (like you suggested) and rebooted the VM the permissions were back to as they were before (no group perms, etc.).

What logs should I look at for udev?

[marmarek]

The file is owned by qubes-utils package. I recommend reinstalling this package (sudo dnf reinstall qubes-utils) to make sure possibly other removed files are restored. But for this, you need network...
You can copy the file from other VM (clipboard should work), then reboot.

[marmarek]

The file is owned by qubes-utils package. I recommend reinstalling this package (sudo dnf reinstall qubes-utils) to make sure possibly other removed files are restored. But for this, you need network...
You can copy the file from other VM (clipboard should work), then reboot.

[esote]

@marmarek I've copied over 99-qubes-block.rules, 99-qubes-misc.rules, 99-qubes-usb.rules, and 60-ffado.rules (because it was also missing when compared with other VMs, looks like it is related to audio stuff).

I am now able to launch applications from the menu perfectly.

However, the VM still does not have networking and the memory still appears to be capped at the initial memory.

journalctl -r shows the following errors:

pulseaudio[1167]: sink cork req state =2, now state=1
pulseaudio[1167]: source cork req state =2, now state=1
pulseaudio[1167]: source cork req state =1, now state=-2
pulseaudio[1167]: sink cork req state =1, now state=-2
pulseaudio[1167]: rec libvchan_fd_for_select=23, ctrl=0x5ced9b27b000
pulseaudio[1167]: play libvchan_fd_for_select=19, ctrl=0x5ced9b27aef0
pulseaudio[1167]: vchan module loading

But I am getting those errors on all my VMs, all of which have correct networking & memory management, so I do not think those errors are to blame.

[esote]

@marmarek I've copied over 99-qubes-block.rules, 99-qubes-misc.rules, 99-qubes-usb.rules, and 60-ffado.rules (because it was also missing when compared with other VMs, looks like it is related to audio stuff).

I am now able to launch applications from the menu perfectly.

However, the VM still does not have networking and the memory still appears to be capped at the initial memory.

journalctl -r shows the following errors:

pulseaudio[1167]: sink cork req state =2, now state=1
pulseaudio[1167]: source cork req state =2, now state=1
pulseaudio[1167]: source cork req state =1, now state=-2
pulseaudio[1167]: sink cork req state =1, now state=-2
pulseaudio[1167]: rec libvchan_fd_for_select=23, ctrl=0x5ced9b27b000
pulseaudio[1167]: play libvchan_fd_for_select=19, ctrl=0x5ced9b27aef0
pulseaudio[1167]: vchan module loading

But I am getting those errors on all my VMs, all of which have correct networking & memory management, so I do not think those errors are to blame.

[marmarek]

See logs related to qubesdb service. Also, check service status (systemctl status qubes-db).

[marmarek]

See logs related to qubesdb service. Also, check service status (systemctl status qubes-db).

[marmarek]

And generally, check logs for errors (use sudo journalctl to see all messages).

[marmarek]

And generally, check logs for errors (use sudo journalctl to see all messages).

[esote]

@marmarek qubes-db.service is active with no errors.

For the code VM, there are not many logs under /var/log/, and none that I can find that related to qubesdb (no logs at all under /var/log/qubes, and all files are empty under /var/log/xen/).

(sudo) journalctl has a lot of auditing messages and info about command failures (like sync-clock) due to lack of networking.

From dom0, /var/log/qubes/qubesdb.code.log just contains a bunch of repeating messages:

...
reconnecting
vchan closed
vchan closed
reconnecting
vchan closed
vchan closed
...

[esote]

@marmarek qubes-db.service is active with no errors.

For the code VM, there are not many logs under /var/log/, and none that I can find that related to qubesdb (no logs at all under /var/log/qubes, and all files are empty under /var/log/xen/).

(sudo) journalctl has a lot of auditing messages and info about command failures (like sync-clock) due to lack of networking.

From dom0, /var/log/qubes/qubesdb.code.log just contains a bunch of repeating messages:

...
reconnecting
vchan closed
vchan closed
reconnecting
vchan closed
vchan closed
...

[marmarek]

If qubes-db.service is running, can you execute qubesdb-read /name (should output vm name)?

[marmarek]

If qubes-db.service is running, can you execute qubesdb-read /name (should output vm name)?

[esote]

@marmarek Yes:

[user@code ~]$ qubesdb-read /name
code

[esote]

@marmarek Yes:

[user@code ~]$ qubesdb-read /name
code

[marmarek]

Ok, so the network problem is somewhere else. For now lets try to bring it manually:

sudo INTERFACE=eth0 /usr/lib/qubes/setup-ip

(it will output some errors when called manually, but should configure network interface anyway)

[marmarek]

Ok, so the network problem is somewhere else. For now lets try to bring it manually:

sudo INTERFACE=eth0 /usr/lib/qubes/setup-ip

(it will output some errors when called manually, but should configure network interface anyway)

[esote]

It appears I do not have setup-ip in /usr/lib/qubes/ ...

[user@code ~]$ sudo INTERFACE=eth0 /usr/lib/qubes/setup-ip
sudo: /usr/lib/qubes/setup-ip: command not found
[user@code ~]$ ls /usr/lib/qubes
close-window     qrexec-agent                    qubes-trigger-sync-appmenus.sh  upgrades-status-notify
icon-sender      qrexec_client_vm                qvm-copy-to-vm.kde              usb-export
init             qrexec-client-vm                qvm-move-to-vm.kde              usb-import
prepare-suspend  qrun-in-vm                      resize-rootfs                   vm-file-editor
qfile-agent      qubes-download-dom0-updates.sh  tar2qfile                       xdg-icon
qfile-unpacker   qubes-rpc-multiplexer           update-proxy-configs
qopen-in-vm      qubes-sync-clock                upgrades-installed-check

In fact, when comparing it with another VM, it appears I am missing 18 files from /usr/lib/qubes/...

[esote]

It appears I do not have setup-ip in /usr/lib/qubes/ ...

[user@code ~]$ sudo INTERFACE=eth0 /usr/lib/qubes/setup-ip
sudo: /usr/lib/qubes/setup-ip: command not found
[user@code ~]$ ls /usr/lib/qubes
close-window     qrexec-agent                    qubes-trigger-sync-appmenus.sh  upgrades-status-notify
icon-sender      qrexec_client_vm                qvm-copy-to-vm.kde              usb-export
init             qrexec-client-vm                qvm-move-to-vm.kde              usb-import
prepare-suspend  qrun-in-vm                      resize-rootfs                   vm-file-editor
qfile-agent      qubes-download-dom0-updates.sh  tar2qfile                       xdg-icon
qfile-unpacker   qubes-rpc-multiplexer           update-proxy-configs
qopen-in-vm      qubes-sync-clock                upgrades-installed-check

In fact, when comparing it with another VM, it appears I am missing 18 files from /usr/lib/qubes/...

[esote]

@marmarek Should I just copy setup-ip from a VM where networking works?

[esote]

@marmarek Should I just copy setup-ip from a VM where networking works?

[marmarek]

Yes.
I wonder what other files are missing...

[marmarek]

Yes.
I wonder what other files are missing...

[marmarek]

The above list looks like the whole qubes-core-agent-networking package got removed.

[marmarek]

The above list looks like the whole qubes-core-agent-networking package got removed.

[esote]

I have networking, after copying setup-ip I am able to ping websites! You're right, sudo dnf list installed | grep qubes-core-agent-networking returns nothing.

What should I do next, to fix this whole mess?

[esote]

I have networking, after copying setup-ip I am able to ping websites! You're right, sudo dnf list installed | grep qubes-core-agent-networking returns nothing.

What should I do next, to fix this whole mess?

[esote]

@marmarek This output may be helpful, if I am missing anything else...

[user@code qubes]$ sudo dnf list installed | grep -e qubes -e xen -e kernel
kernel.x86_64                          4.15.10-200.fc26                 @updates
kernel.x86_64                          4.15.17-200.fc26                 @updates
kernel-core.x86_64                     4.15.17-200.fc26                 @System 
kernel-devel.x86_64                    4.15.10-200.fc26                 @updates
kernel-devel.x86_64                    4.15.17-200.fc26                 @updates
kernel-headers.x86_64                  4.15.17-200.fc26                 @System 
kernel-modules.x86_64                  4.15.17-200.fc26                 @System 
python3-dnf-plugins-qubes-hooks.x86_64 4.0.24-1.fc26                    @qubes-builder-vm-r4.0-current-testing
qubes-core-agent-qrexec.x86_64         4.0.24-1.fc26                    @qubes-builder-vm-r4.0-current-testing
qubes-db.x86_64                        4.0.5-1.fc26                     @qubes-builder-vm-r4.0-current-testing
qubes-gui-agent.x86_64                 4.0.9-1.fc26                     @qubes-builder-vm-r4.0-current-testing
qubes-menus.noarch                     4.0.12-1.fc26                    @System 
qubes-mgmt-salt-vm-connector.noarch    4.0.8-1.fc26                     @System 
qubes-utils-libs.x86_64                4.0.17-1.fc26                    @qubes-builder-vm-r4.0-current-testing
qubes-vm-dependencies.noarch           4.0.1-1.fc26                     @qubes-builder-vm-r4.0-current-testing
texlive-l3kernel.noarch                6:svn41246-33.fc26.2             @fedora 
xen-libs.x86_64                        2001:4.8.3-3.fc26                @qubes-vm-r4.0-current
xen-licenses.x86_64                    2001:4.8.3-3.fc26                @qubes-vm-r4.0-current
xen-qubes-vm.x86_64                    2001:4.8.3-3.fc26                @qubes-vm-r4.0-current

(the kernel package is 4.15, but uname -r returns 4.14.18-1.pvops.qubes.x86_64, no worries there I don't think)

[esote]

@marmarek This output may be helpful, if I am missing anything else...

[user@code qubes]$ sudo dnf list installed | grep -e qubes -e xen -e kernel
kernel.x86_64                          4.15.10-200.fc26                 @updates
kernel.x86_64                          4.15.17-200.fc26                 @updates
kernel-core.x86_64                     4.15.17-200.fc26                 @System 
kernel-devel.x86_64                    4.15.10-200.fc26                 @updates
kernel-devel.x86_64                    4.15.17-200.fc26                 @updates
kernel-headers.x86_64                  4.15.17-200.fc26                 @System 
kernel-modules.x86_64                  4.15.17-200.fc26                 @System 
python3-dnf-plugins-qubes-hooks.x86_64 4.0.24-1.fc26                    @qubes-builder-vm-r4.0-current-testing
qubes-core-agent-qrexec.x86_64         4.0.24-1.fc26                    @qubes-builder-vm-r4.0-current-testing
qubes-db.x86_64                        4.0.5-1.fc26                     @qubes-builder-vm-r4.0-current-testing
qubes-gui-agent.x86_64                 4.0.9-1.fc26                     @qubes-builder-vm-r4.0-current-testing
qubes-menus.noarch                     4.0.12-1.fc26                    @System 
qubes-mgmt-salt-vm-connector.noarch    4.0.8-1.fc26                     @System 
qubes-utils-libs.x86_64                4.0.17-1.fc26                    @qubes-builder-vm-r4.0-current-testing
qubes-vm-dependencies.noarch           4.0.1-1.fc26                     @qubes-builder-vm-r4.0-current-testing
texlive-l3kernel.noarch                6:svn41246-33.fc26.2             @fedora 
xen-libs.x86_64                        2001:4.8.3-3.fc26                @qubes-vm-r4.0-current
xen-licenses.x86_64                    2001:4.8.3-3.fc26                @qubes-vm-r4.0-current
xen-qubes-vm.x86_64                    2001:4.8.3-3.fc26                @qubes-vm-r4.0-current

(the kernel package is 4.15, but uname -r returns 4.14.18-1.pvops.qubes.x86_64, no worries there I don't think)

[marmarek]

Try to install qubes-vm-recommended, it should pull all the rest useful packages. See here for description of some: https://www.qubes-os.org/doc/templates/fedora-minimal/

[marmarek]

Try to install qubes-vm-recommended, it should pull all the rest useful packages. See here for description of some: https://www.qubes-os.org/doc/templates/fedora-minimal/

[esote]

@marmarek

[user@code qubes]$ sudo dnf install qubes-vm-recommended
Last metadata expiration check: 0:08:42 ago on Sun 06 May 2018 08:47:22 PM CDT.
Error: 
 Problem: conflicting requests
  - nothing provides qubes-input-proxy-sender needed by qubes-vm-recommended-4.0.1-1.fc26.noarch

Is there a package called qubes-input-proxy-sender I need beforehand? (sudo dnf info qubes-input-proxy-sender returns "no matching packages")

[esote]

@marmarek

[user@code qubes]$ sudo dnf install qubes-vm-recommended
Last metadata expiration check: 0:08:42 ago on Sun 06 May 2018 08:47:22 PM CDT.
Error: 
 Problem: conflicting requests
  - nothing provides qubes-input-proxy-sender needed by qubes-vm-recommended-4.0.1-1.fc26.noarch

Is there a package called qubes-input-proxy-sender I need beforehand? (sudo dnf info qubes-input-proxy-sender returns "no matching packages")

[marmarek]

Try again now (you may need --refresh option).

[marmarek]

Try again now (you may need --refresh option).

[esote]

It is now complaining of qubes-pdf-converter in the same type of message, haha

[esote]

It is now complaining of qubes-pdf-converter in the same type of message, haha

[marmarek]

Try again and tell me what is missing this time...

[marmarek]

Try again and tell me what is missing this time...

[esote]

thunderbird-qubes

[esote]

thunderbird-qubes

[marmarek]

Now?

[marmarek]

Now?

[esote]

It works! I've installed the package.

Upon seeing the packages being installed by qubes-vm-recommended, I believe I've found the cause of all this.
Some time ago, I removed the thunderbird package (thinking I didn't need the client for anything I was working on). In doing so, it automatically removed the Qubes networking & some peripheral packages (perhaps because dnf thought they were "no longer needed"?). I suppose since I do not use networking on this VM too often, I did not notice for a while.

Am I safe to restart the VM, or is there more to do / check?

[esote]

It works! I've installed the package.

Upon seeing the packages being installed by qubes-vm-recommended, I believe I've found the cause of all this.
Some time ago, I removed the thunderbird package (thinking I didn't need the client for anything I was working on). In doing so, it automatically removed the Qubes networking & some peripheral packages (perhaps because dnf thought they were "no longer needed"?). I suppose since I do not use networking on this VM too often, I did not notice for a while.

Am I safe to restart the VM, or is there more to do / check?

[marmarek]

Restart should be ok now. As for automatic removal by dnf - there is an option in /etc/dnf/dnf.conf (clean_requirements_on_remove).

[marmarek]

Restart should be ok now. As for automatic removal by dnf - there is an option in /etc/dnf/dnf.conf (clean_requirements_on_remove).

[esote]

Yes, I may set that to False for now, or even better I should really look at the packages being removed by dnf before agreeing to the removal... as a matter of habit

@marmarek Thanks so much for taking the time to help me with this issue, and thanks for all your work on the Qubes project!

[esote]

Yes, I may set that to False for now, or even better I should really look at the packages being removed by dnf before agreeing to the removal... as a matter of habit

@marmarek Thanks so much for taking the time to help me with this issue, and thanks for all your work on the Qubes project!