Updates fail when update proxy VM not already running

[tasket]

Qubes OS version:

R4.0

Affected component(s):

Template updates

---

Steps to reproduce the behavior:

Initiate update procedure for any template whose update proxy is not already running. For example, whonix-ws when sys-whonix is not running.

Expected behavior:

Proxy comes online before update begins, or update waits for proxy to come online.

Actual behavior:

Many 'unable to retrieve' type errors until the update process fails. Update must be coordinated manually instead.

General notes:

The method by which the template's network dependency is satisfied is unlike the normal handling of an assigned netvm; the asynchronous nature doesn't lend itself to predictable updates with normal update clients such as apt, dnf, etc.

---

Related issues:

[marmarek]

Does it happen for non-whonix update proxy VMs too?

[marmarek]

Does it happen for non-whonix update proxy VMs too?

[adrelanos]

Yes, because qvm-start logic does not analyze /etc/qubes-rpc/policy/qubes.UpdatesProxy.

---

Qubes-Whonix 14 addresses this and has code to wait which seems to work quite well.

https://github.com/Whonix/qubes-whonix/blob/master/etc/uwt.d/40_qubes.conf

But that is not pretty. I would prefer if Qubes would make update proxy VM start before the TemplateVM starts as it was the case in Qubes R3.2.

In case you want to test:

• https://www.whonix.org/wiki/Qubes/Install/Testing
• https://forums.whonix.org/t/whonix-14-release-blockers-status-of-whonix-14-development

---

Related:
#3412

[adrelanos]

Yes, because qvm-start logic does not analyze /etc/qubes-rpc/policy/qubes.UpdatesProxy.

---

Qubes-Whonix 14 addresses this and has code to wait which seems to work quite well.

https://github.com/Whonix/qubes-whonix/blob/master/etc/uwt.d/40_qubes.conf

But that is not pretty. I would prefer if Qubes would make update proxy VM start before the TemplateVM starts as it was the case in Qubes R3.2.

In case you want to test:

• https://www.whonix.org/wiki/Qubes/Install/Testing
• https://forums.whonix.org/t/whonix-14-release-blockers-status-of-whonix-14-development

---

Related:
#3412

[rustybird]

For non-Whonix templates, I only seem to have this problem when my system is under high load. I'd guess that dnf and maybe apt-get have some sort of (configurable?) minimum throughput or read timeout, so if the updates proxy VM takes too long to start, they give up on the connection. Without retrying - good thing that internet connections are 100% reliable.

I would prefer if Qubes would make update proxy VM start before the TemplateVM starts as it was the case in Qubes R3.2.

Could be pretty rough if you're starting the template for reasons other than installing software, e.g. to edit a configuration file. In the worst case, this would cause sys-net -> sys-firewall -> sys-whonix startup, maybe even a fourth VM if the updates proxy is running in a separate VM. People seemed to dislike that for clock sync (#3588).

For my Whonix templates, I've opted to disable the Whonix safety mechanism by enabling their whonix-secure-proxy qvm-service.

[rustybird]

For non-Whonix templates, I only seem to have this problem when my system is under high load. I'd guess that dnf and maybe apt-get have some sort of (configurable?) minimum throughput or read timeout, so if the updates proxy VM takes too long to start, they give up on the connection. Without retrying - good thing that internet connections are 100% reliable.

I would prefer if Qubes would make update proxy VM start before the TemplateVM starts as it was the case in Qubes R3.2.

Could be pretty rough if you're starting the template for reasons other than installing software, e.g. to edit a configuration file. In the worst case, this would cause sys-net -> sys-firewall -> sys-whonix startup, maybe even a fourth VM if the updates proxy is running in a separate VM. People seemed to dislike that for clock sync (#3588).

For my Whonix templates, I've opted to disable the Whonix safety mechanism by enabling their whonix-secure-proxy qvm-service.

[adrelanos]

Could be pretty rough if you're starting the template for reasons other than installing software, [...]

Fair point. However, Qubes automatically starting dnf / apt-get for updating negates this.

[adrelanos]

Could be pretty rough if you're starting the template for reasons other than installing software, [...]

Fair point. However, Qubes automatically starting dnf / apt-get for updating negates this.

[rustybird]

However, Qubes automatically starting dnf / apt-get for updating negates this.

If you're opening the template for a quick edit, you have a leisurely ;) 5 minutes until qubes-update-check.timer kicks in. (I usually just disable the qubes-update-check qvm-service.)

[rustybird]

However, Qubes automatically starting dnf / apt-get for updating negates this.

If you're opening the template for a quick edit, you have a leisurely ;) 5 minutes until qubes-update-check.timer kicks in. (I usually just disable the qubes-update-check qvm-service.)

[tasket]

In addition to whonix templates, a regular template will likely fail if its configured to redirect to an "update VM" that is also dependent on another netvm and they both have to start. For example, a logical combination is:
Mytemplate >> sys-update --> sys-vpn.

For the time being, I'm playing with parsing the qubes.UpdatesProxy policy from my qubes4-multi-update.py script -- though I wish I had a general solution that worked like Whonix 14. I'd rather have some way for the template to test the update proxy connection (hint...).

[tasket]

In addition to whonix templates, a regular template will likely fail if its configured to redirect to an "update VM" that is also dependent on another netvm and they both have to start. For example, a logical combination is:
Mytemplate >> sys-update --> sys-vpn.

For the time being, I'm playing with parsing the qubes.UpdatesProxy policy from my qubes4-multi-update.py script -- though I wish I had a general solution that worked like Whonix 14. I'd rather have some way for the template to test the update proxy connection (hint...).

[tasket]

FWIW, I currently cannot reproduce the problem with regular templates after 3 tries (starting 2 upstream VMs each time).

The problem persists for Whonix, however. I am trying a workaround to trigger the update VM to start and then wait for it, like:

curl -s --connect-timeout 30 localhost:8082 >/dev/null
sleep 10s
# start update process...

[tasket]

FWIW, I currently cannot reproduce the problem with regular templates after 3 tries (starting 2 upstream VMs each time).

The problem persists for Whonix, however. I am trying a workaround to trigger the update VM to start and then wait for it, like:

curl -s --connect-timeout 30 localhost:8082 >/dev/null
sleep 10s
# start update process...

[adrelanos]

Also as far as I understand in Qubes R4, templates are started after being downloaded so they can process /etc/qubes/post-install.d/ (qvm-features-request mechanism) and then shut down. This needs to be kept in mind.

• https://github.com/QubesOS/qubes-core-agent-linux/blob/master/qubes-rpc/qubes.PostInstall
• https://github.com/QubesOS/qubes-core-agent-linux/tree/master/post-install.d

[adrelanos]

Also as far as I understand in Qubes R4, templates are started after being downloaded so they can process /etc/qubes/post-install.d/ (qvm-features-request mechanism) and then shut down. This needs to be kept in mind.

• https://github.com/QubesOS/qubes-core-agent-linux/blob/master/qubes-rpc/qubes.PostInstall
• https://github.com/QubesOS/qubes-core-agent-linux/tree/master/post-install.d