Salt-based updates fail on Debian and Whonix templates when using a Debian-based mgmt VM

[airelemental]

Qubes OS version
R4.0 with latest updates applied

Affected component(s) or functionality
qubes-update-gui
qubesctl

Brief summary
qubes-update-gui and qubesctl fail to update debian-10 and whonix-15 templates, with obscure error.

How Reproducible
Always

To Reproduce

1. use qubes-update-gui to upgrade any debian or whonix template

OR

1. sudo qubesctl --skip-dom0 --targets=debian-10-test --show-output state.sls update.qubes-vm

Expected behavior

green check mark

Actual behavior
red cross
debian-10-test: ERROR (exception list index out of range)

A workaround
disp-mgmt console says:

user@disp-mgmt-debian-10-test:~$ sudo .journalctl -b | grep -i salt
May 26 11:27:07 disp-mgmt-debian-10-test qrexec-agent[504]: executed user:QUBESRPC qubes.SaltLinuxVM dom0 pid 584
May 26 11:27:07 disp-mgmt-debian-10-test sudo[585]:     user : TTY=unknown ; PWD=/home/user ; USER=root ; COMMAND=/etc/qubes-rpc/qubes.SaltLinuxVM
May 26 11:27:07 disp-mgmt-debian-10-test qubes.SaltLinuxVM-dom0[593]: sed: can't read /usr/lib/python3*/site-packages/salt/utils/jinja.py: No such file or directory

In disp-mgmt VM's /etc/qubes-rpc/qubes.SaltLinuxVM, there is a sed at the bottom that runs on jinga.py inside site-packages which doesn't exist in debian-10 and whonix-ws-15.

There's a dist-packages/ though, and if I replace "site-packages" by "dist-packages" in the sed, the upgrade seems to work.

Thanks to oush9 on qubes:matrix.org for troubleshooting help.

[andrewdavidwong]

A Fedora-based mgmt VM can manage both Fedora and Debian VMs, but a Debian-based mgmt VM can manage only other Debian VMs, not Fedora VMs. Could this be your problem?

This applies to 4.0, but I think it's already fixed in 4.1. @DemiMarie, can you confirm, and you can you provide the number for that issue?

[DemiMarie]

This applies to 4.0, but I think it's already fixed in 4.1. @DemiMarie, can you confirm, and you can you provide the number for that issue?

Honestly I am not sure.

[airelemental]

A Fedora-based mgmt VM can manage both Fedora and Debian VMs, but a Debian-based mgmt VM can manage only other Debian VMs, not Fedora VMs. Could this be your problem?

I use debian-based VMs for everything...

Maybe that is why it hasn't been reported before - most people use fedora-based mgmt VM, which might have site-packages instead of dist-packages.

[DemiMarie]

A Fedora-based mgmt VM can manage both Fedora and Debian VMs, but a Debian-based mgmt VM can manage only other Debian VMs, not Fedora VMs. Could this be your problem?

I use debian-based VMs for everything...

Maybe that is why it hasn't been reported before - most people use fedora-based mgmt VM, which might have site-packages instead of dist-packages.

That would certainly do it! I will fix this.

[DemiMarie]

This applies to 4.0, but I think it's already fixed in 4.1. @DemiMarie, can you confirm, and you can you provide the number for that issue?

Honestly I am not sure.

To elaborate: the problem is that Fedora has a newer version of Python, which is incompatible with the older version of Salt in Debian. Fixing this would likely require that Qubes OS either ship its own version of Salt (overriding the distribution package), or that each non-minimal TemplateVM package provide a ManagementVM based on that TemplateVM (which is what I recommend).

[andrewdavidwong]

This applies to 4.0, but I think it's already fixed in 4.1. @DemiMarie, can you confirm, and you can you provide the number for that issue?

Honestly I am not sure.

To elaborate: the problem is that Fedora has a newer version of Python, which is incompatible with the older version of Salt in Debian. Fixing this would likely require that Qubes OS either ship its own version of Salt (overriding the distribution package), or that each non-minimal TemplateVM package provide a ManagementVM based on that TemplateVM (which is what I recommend).

Can you confirm whether this is fixed in 4.1? (I thought I recall hearing that from you, but now I'm not sure.)

If so, then the question is what, if anything, to do about it in 4.0. One option is just to document the fact that a Fedora TemplateVM should be used for the mgmt VM in 4.0. @marmarek, thoughts?

[unman]

On Wed, May 26, 2021 at 02:19:19PM -0700, Andrew David Wong wrote: > > > This applies to 4.0, but I think it's already fixed in 4.1. @DemiMarie, can you confirm, and you can you provide the number for that issue? > > > > > > Honestly I am not sure. > > To elaborate: the problem is that Fedora has a newer version of Python, which is incompatible with the older version of Salt in Debian. Fixing this would likely require that Qubes OS either ship its own version of Salt (overriding the distribution package), or that each non-minimal TemplateVM package provide a ManagementVM based on that TemplateVM (which is what I recommend). Can you confirm whether this is fixed in 4.1? (I thought I recall hearing that from you, but now I'm not sure.) If so, then the question is what, if anything, to do about it in 4.0. One option is just to document the fact that a Fedora TemplateVM should be used for the `mgmt` VM in 4.0. @marmarek, thoughts?

It's worth pointing out that airelemental neglected a crucial fact - that they are using update-candidates. So the system has worked well - something in testing has been found to be wrong, and is now being fixed. It has no impact on the main release. Two things: 1. There should be some way of flagging these cases - so that reporters and other users know they relate to "testing". 2. There should be more testing *before* updated packages are pushed to testing. At present, it seems restricted to "does it build". ( I often find that changes break Ubuntu, and older Debian.) I know this is difficult with a small team: it isn't impossible.

[andrewdavidwong]

It's worth pointing out that airelemental neglected a crucial fact - that they are using update-candidates.
[...]

1. There should be some way of flagging these cases - so that reporters and other users know they relate to "testing".

That's why the very first field in the issue template is "Qubes OS version."

In this case, @airelemental wrote:

Qubes OS version
R4.0 with latest updates applied

Was this a lie?

[unman]

On Thu, May 27, 2021 at 06:19:36AM -0700, Andrew David Wong wrote: > It's worth pointing out that airelemental neglected a crucial fact - that they are using update-candidates. > [...] > 1. There should be some way of flagging these cases - so that reporters and other users know they relate to "testing". That's why the very first field in the issue template is "Qubes OS version." In this case, @airelemental wrote: > **Qubes OS version** > R4.0 with latest updates applied Was this a lie?

Certainly not a lie - misleading. It would depend on whether you interpret "latest" as "latest updates" or "latest candidates for updates". These are different things. Flagging such issues would enable devs to block package transition to "updates": otherwise *many* users would encounter the problem.

[brendanhoar]

Personally I do try to remember to indicate that I am using the testing repo when filling out a QubesOS bug report. Perhaps a reference to that can be added to the template in the first section for QubesOS version, e.g. "If using testing, security-testing or other non-default repos, please indicate which one(s) here as well." and then Andrew can tag accordingly.

[unman]

On Thu, May 27, 2021 at 07:12:20AM -0700, Brendan Hoar wrote: Personally I do try to remember to indicate that I am using the testing repo when filling out a QubesOS bug report. Perhaps a reference to that can be added to the template in the first section for QubesOS version, e.g. "If using testing, security-testing or other non-default repos, please indicate which one(s) here as well."

Yes, this would be good, although many users wont remember after they have enabled those repositories. Having a Label for Testing/Proposed_update would, I think, be useful, since there is some immediacy required in dealing with those issues.

[andrewdavidwong]

Personally I do try to remember to indicate that I am using the testing repo when filling out a QubesOS bug report. Perhaps a reference to that can be added to the template in the first section for QubesOS version, e.g. "If using testing, security-testing or other non-default repos, please indicate which one(s) here as well." and then Andrew can tag accordingly.

Thanks for the feedback. I've updated the bug report issue template to add sections regarding testing: 0ee2cc3. Hopefully this helps.

Flagging such issues would enable devs to block package transition to "updates": otherwise many users would encounter the problem.

We actually already have a procedure in place for providing feedback on packages in testing via the updates-status repo. I wasn't sure exactly how best to integrate this into the bug report issue template, but I figure we want to avoid a situation in which someone reports a bug here without indicating anything in updates-status or without linking together to their bug report here with their comment in updates-status, so I added an extra section specifically for this. (CC @marmarek)

Having a Label for Testing/Proposed_update would, I think, be useful, since there is some immediacy required in dealing with those issues.

Unsure what label text would be most perspicuous here. testing is likely to be misused for things like writing unit tests and other work related to automated testing. Since we already have the updates-status repo, we could use that same name for this label, but people who don't know about updates-status might not make that connection. updates-testing might be confusing, given that we already have C: updates. It could be misinterpreted as testing the update system itself, but I suppose this is unlikely and the least of three evils. I'll go with this for now.

[unman]

On Thu, May 27, 2021 at 09:25:22PM -0700, Andrew David Wong wrote: > Personally I do try to remember to indicate that I am using the testing repo when filling out a QubesOS bug report. Perhaps a reference to that can be added to the template in the first section for QubesOS version, e.g. "If using testing, security-testing or other non-default repos, please indicate which one(s) here as well." and then Andrew can tag accordingly. Thanks for the feedback. I've updated the bug report issue template to add sections regarding testing: 0ee2cc3. Hopefully this helps. > Flagging such issues would enable devs to block package transition to "updates": otherwise *many* users would encounter the problem. We actually already have a procedure in place for [providing feedback on packages in testing via the `updates-status` repo](https://www.qubes-os.org/doc/testing/#providing-feedback). I wasn't sure exactly how best to integrate this into the bug report issue template, but I figure we want to avoid a situation in which someone reports a bug here without indicating anything in `updates-status` or without linking together to their bug report here with their comment in `updates-status`, so I added an extra section specifically for this. (CC @marmarek) > Having a Label for Testing/Proposed_update would, I think, be useful, since there is some immediacy required in dealing with those issues. Unsure what label text would be most perspicuous here. `testing` is likely to be misused for things like writing unit tests and other work related to automated testing. Since we already have the `updates-status` repo, we could use that same name for this label, but people who don't know about `updates-status` might not make that connection. `updates-testing` might be confusing, given that we already have `C: updates`. It could be misinterpreted as testing the update system itself, but I suppose this is unlikely and the least of three evils. I'll go with this for now.

If I knew how to give a thumbs up, I would do so.

[ThomasWaldmann]

I also ran into this and worked around it by:

Menu -> System Tools -> Qubes Template Manager

default-mgmt-dvm: switched to fedora (had debian there)

[andrewdavidwong]

@ThomasWaldmann wrote:

I also ran into this

Are you also on 4.0 with testing repos enabled? If so, which ones? (@airelemental, I would also like to ask these same questions of you.)

---

@unman wrote:

something in testing has been found to be wrong, and is now being fixed. It has no impact on the main release.

To clarify, by "main release" you mean "stable release (4.0) without any testing repos enabled"?

---

@DemiMarie wrote:

Fixing this would likely require that Qubes OS either ship its own version of Salt (overriding the distribution package), or that each non-minimal TemplateVM package provide a ManagementVM based on that TemplateVM (which is what I recommend).

To be clear, you're saying that one of these fixes would likely be required for fixing this in 4.0, right?

Regarding your second/recommended proposal, I foresee a few challenges:

• Users of minimal templates still wouldn't get the fix. (Does this just get documented and considered acceptable per the health warning?)
• It wouldn't be obvious to users why a ManagementVM is being created when installing a template.
• Users might remove the automatically-created ManagementVMs, not realizing that they're needed or why.
• This would only solve the problem for new RPM-installed templates, not for existing templates. (Unless that's okay because only installations with certain testing repos enabled are affected?)

[qubesos-bot]

Automated announcement from builder-github

The component mgmt-salt (including package qubes-mgmt-salt-4.1.12-1.fc32) has been pushed to the r4.1 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-mgmt-salt-4.1.12-1.fc32 has been pushed to the r4.1 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-mgmt-salt_4.0.26-1+deb10u1 has been pushed to the r4.0 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The component mgmt-salt (including package qubes-mgmt-salt-4.0.26-1.fc32) has been pushed to the r4.0 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-mgmt-salt-4.0.26-1.fc25 has been pushed to the r4.0 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update