quickly restarting VM leads to race condition, VM startup and keyboard issues

[adrelanos]

By switching Qubes VM Manger Firewall tab settings back and forth, I managed to end up with an unstartable VM.

qvm-start my-whonix-ws

--> Creating volatile image: /var/lib/qubes/appvms/my-whonix-ws/volatile.img...
--> Loading the VM (type = AppVM)...
--> Starting Qubes DB...
--> Setting Qubes DB info for the VM...
--> Updating firewall rules...
Traceback (most recent call last):
  File "/usr/bin/qvm-start", line 125, in <module>
    main()
  File "/usr/bin/qvm-start", line 109, in main
    xid = vm.start(verbose=options.verbose, preparing_dvm=options.preparing_dvm, start_guid=not options.noguid, notify_function=tray_notify_generic if options.tray else None)
  File "/usr/lib64/python2.7/site-packages/qubes/modules/000QubesVm.py", line 1788, in start
    netvm.write_iptables_qubesdb_entry()
  File "/usr/lib64/python2.7/site-packages/qubes/modules/006QubesProxyVm.py", line 117, in write_iptables_qubesdb_entry
    self.qdb.rm("/qubes-iptables-domainrules/")
  File "/usr/lib64/python2.7/site-packages/qubes/modules/000QubesVm.py", line 689, in qdb
    self._qdb_connection = QubesDB(self.name)
qubes.qdb.Error: (2, 'No such file or directory')

Can you make head or tail of this?

[adrelanos]

Toggling firewall settings of another AppVM behind sys-firewall (not sys-whonix) works as workaround.

[adrelanos]

Toggling firewall settings of another AppVM behind sys-firewall (not sys-whonix) works as workaround.

[adrelanos]

Or not. It only makes the "No such file or directory" error in Qubes VM manager temporarily go away. Starting the VM from Qubes VM Manager or command line is still failing.

[adrelanos]

Or not. It only makes the "No such file or directory" error in Qubes VM manager temporarily go away. Starting the VM from Qubes VM Manager or command line is still failing.

[marmarek]

Apparently QubesDB daemon for sys-firewall (or whatever proxyvm you
have there) have died. Check /var/log/qubes/qubesdb.sys-firewall.log.
Check also process list - you should have one qubesdb-daemon process for
each running VM (including dom0).

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

Apparently QubesDB daemon for sys-firewall (or whatever proxyvm you
have there) have died. Check /var/log/qubes/qubesdb.sys-firewall.log.
Check also process list - you should have one qubesdb-daemon process for
each running VM (including dom0).

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[adrelanos]

/var/log/qubes/qubesdb.sys-firewall.log:

vchan closed
vchan reconnecting
vchan closed

sys-firewall has qubesdb-daemon running. Also dom0 has a qubesdb-daemon running for sys-firewall.

[adrelanos]

/var/log/qubes/qubesdb.sys-firewall.log:

vchan closed
vchan reconnecting
vchan closed

sys-firewall has qubesdb-daemon running. Also dom0 has a qubesdb-daemon running for sys-firewall.

[marmarek]

And the sys-firewall is the netvm of that AppVM, right? Can you access
its QubesDB from dom0 (qubesdb-list -d sys-firewall /)?

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

And the sys-firewall is the netvm of that AppVM, right? Can you access
its QubesDB from dom0 (qubesdb-list -d sys-firewall /)?

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[adrelanos]

And the sys-firewall is the netvm of that AppVM, right?

Yes.

Can you access its QubesDB from dom0 (qubesdb-list -d sys-firewall /)?

Yes, works. Lists everything.

[adrelanos]

And the sys-firewall is the netvm of that AppVM, right?

Yes.

Can you access its QubesDB from dom0 (qubesdb-list -d sys-firewall /)?

Yes, works. Lists everything.

[adrelanos]

And the sys-firewall is the netvm of that AppVM, right?

Ehrm. No. Sorry. my-whonix-ws's netvm is sys-whonix.

[adrelanos]

And the sys-firewall is the netvm of that AppVM, right?

Ehrm. No. Sorry. my-whonix-ws's netvm is sys-whonix.

[adrelanos]

sys-whonix also has qubesdb-daemon running.

[adrelanos]

sys-whonix also has qubesdb-daemon running.

[adrelanos]

There is another issue in sys-whonix at the moment. But it's not related to Whonix. It also happened in Debian VMs to be earlier. The keyboard layout is messed up. querty even though I usually use qwertz. I had this before. It seems to be a race condition. After reboot it works. So the bug I am reporting here is perhaps just a follow up issue of the one I am describing at the moment.

[adrelanos]

There is another issue in sys-whonix at the moment. But it's not related to Whonix. It also happened in Debian VMs to be earlier. The keyboard layout is messed up. querty even though I usually use qwertz. I had this before. It seems to be a race condition. After reboot it works. So the bug I am reporting here is perhaps just a follow up issue of the one I am describing at the moment.

[adrelanos]

qubesdb-read /qubes-keyboard currently fails in sys-whonix. qubesdb-list / works generally, but does not the list the /qubes-keyboard keyword.

[adrelanos]

qubesdb-read /qubes-keyboard currently fails in sys-whonix. qubesdb-list / works generally, but does not the list the /qubes-keyboard keyword.

[marmarek]

Keyboard layout is also passed to the VM through QubesDB, so if there is
a problem with it, keyboard layout also will not be loaded.
Can you access QubesDB of sys-whonix from dom0?

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

Keyboard layout is also passed to the VM through QubesDB, so if there is
a problem with it, keyboard layout also will not be loaded.
Can you access QubesDB of sys-whonix from dom0?

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[adrelanos]

No.

In dom0.

qubesdb-list -d sys-whonix

Failed to connect to sys-whonix daemon

[adrelanos]

No.

In dom0.

qubesdb-list -d sys-whonix

Failed to connect to sys-whonix daemon

[marmarek]

Ok, so here is the problem. Anything interesting in
/var/log/qubes/qubesdb.sys-whonix.log? Do you remember when
sys-whonix was started? Does it match start time of its
qubesdb-daemon process in dom0? Did you started sys-whonix just
after shutting it down (aka restart)?

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

Ok, so here is the problem. Anything interesting in
/var/log/qubes/qubesdb.sys-whonix.log? Do you remember when
sys-whonix was started? Does it match start time of its
qubesdb-daemon process in dom0? Did you started sys-whonix just
after shutting it down (aka restart)?

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

I guess it may be some race condition in starting qubesdb-daemon in
dom0, when the previous one is still running for the same VM (which was
just stopped). I've seen this for Windows VMs, but maybe it also happens
for Linux ones...

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

I guess it may be some race condition in starting qubesdb-daemon in
dom0, when the previous one is still running for the same VM (which was
just stopped). I've seen this for Windows VMs, but maybe it also happens
for Linux ones...

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[adrelanos]

Anything interesting in /var/log/qubes/qubesdb.sys-whonix.log?

/var/log/qubes/qubesdb.sys-whonix.log:

vchan closed
vchan reconnecting
vchan closed

Do you remember when sys-whonix was started?

No.

Did you started sys-whonix just after shutting it down (aka restart)?

Possibly. I often shut it down and restart right after for testing purposes.

[adrelanos]

Anything interesting in /var/log/qubes/qubesdb.sys-whonix.log?

/var/log/qubes/qubesdb.sys-whonix.log:

vchan closed
vchan reconnecting
vchan closed

Do you remember when sys-whonix was started?

No.

Did you started sys-whonix just after shutting it down (aka restart)?

Possibly. I often shut it down and restart right after for testing purposes.

[adrelanos]

Do you have enough info to make this ticket actionable / duplicate of something? Do you want more debug info? Or should I leave it that way for some more time for your consideration?

Just asking, because otherwise I would just try restart sys-whonix or the whole system as workaround.

[adrelanos]

Do you have enough info to make this ticket actionable / duplicate of something? Do you want more debug info? Or should I leave it that way for some more time for your consideration?

Just asking, because otherwise I would just try restart sys-whonix or the whole system as workaround.

[marmarek]

Ok, I think I have all the information needed to fix this. So for now
you can simply restart sys-whonix to have it working again (make sure
that no qubesdb-daemon for it is running before starting it again)

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

Ok, I think I have all the information needed to fix this. So for now
you can simply restart sys-whonix to have it working again (make sure
that no qubesdb-daemon for it is running before starting it again)

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[adrelanos]

Alright. Works now again. And btw I just now also was able to reproduce the qubes-keyboard issue with a Debian VM that I shutdown and restarted quickly which strengthens your hypothesis. (#1241 (comment))

[adrelanos]

Alright. Works now again. And btw I just now also was able to reproduce the qubes-keyboard issue with a Debian VM that I shutdown and restarted quickly which strengthens your hypothesis. (#1241 (comment))

[schnaser]

Has a fix been released for this? It also appears to happen when there are "a bunch" of entries in firewall.xml. I can currently reproduce this 100% by:

1. Create a new VM. Do not start it.
2. Paste the attached firewall.txt file into /var/lib/qubes/appvms/whatever/firewall.xml
3. Attempt to start the VM.
4. Get error

The firewall is nothing special IMO, just a list of the top 50 most popular sites. My banking VM where I first ran into this has 45 entries.

firewall.txt

I notice that the error is slightly different, however so this may well be a separate issue.

error.txt

[schnaser]

Has a fix been released for this? It also appears to happen when there are "a bunch" of entries in firewall.xml. I can currently reproduce this 100% by:

1. Create a new VM. Do not start it.
2. Paste the attached firewall.txt file into /var/lib/qubes/appvms/whatever/firewall.xml
3. Attempt to start the VM.
4. Get error

The firewall is nothing special IMO, just a list of the top 50 most popular sites. My banking VM where I first ran into this has 45 entries.

firewall.txt

I notice that the error is slightly different, however so this may well be a separate issue.

error.txt

[marmarek]

I think the problem with such "large" firewall is rather #1570

[marmarek]

I think the problem with such "large" firewall is rather #1570