Fedora-based management qube causes Salt errors when updating certain templates: `Failed to return clean data... ModuleNotFoundError: No module named 'urllib3.util.ssl_match_hostname'`

[Minimalist73]

How to file a helpful issue

Qubes OS release

4.1

Brief summary

Updating Debian-11 based templates/qubes with salt results in a salt error about urllib3.util.ssl_match_hostname not being found.

Steps to reproduce

• Open a dom0 terminal
• Update with the following command:
  • qubesctl --show-output --skip-dom0 --targets debian-11 state.apply update.qubes-vm

Expected behavior

Template/qubes based on debian-11 update without issues

Actual behavior

Salt return an error:

Traceback (most recent call last):
            File "/var/tmp/.root_dd8a91_salt/salt-call", line 27, in <module>
              salt_call()
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/scripts.py", line 438, in salt_call
              import salt.cli.call
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/cli/call.py", line 3, in <module>
              import salt.cli.caller
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/cli/caller.py", line 12, in <module>
              import salt.channel.client
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/channel/client.py", line 13, in <module>
              import salt.crypt
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/crypt.py", line 26, in <module>
              import salt.payload
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/payload.py", line 12, in <module>
              import salt.loader.context
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/loader/__init__.py", line 23, in <module>
              import salt.utils.event
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/utils/event.py", line 67, in <module>
              import salt.ext.tornado.iostream
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/ext/tornado/iostream.py", line 42, in <module>
              import urllib3.util.ssl_match_hostname
          ModuleNotFoundError: No module named 'urllib3.util.ssl_match_hostname'
          [ERROR   ] An un-handled exception was caught by Salt's global exception handler:
          ModuleNotFoundError: No module named 'urllib3.util.ssl_match_hostname'
          Traceback (most recent call last):
            File "/var/tmp/.root_dd8a91_salt/salt-call", line 27, in <module>
              salt_call()
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/scripts.py", line 438, in salt_call
              import salt.cli.call
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/cli/call.py", line 3, in <module>
              import salt.cli.caller
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/cli/caller.py", line 12, in <module>
              import salt.channel.client
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/channel/client.py", line 13, in <module>
              import salt.crypt
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/crypt.py", line 26, in <module>
              import salt.payload
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/payload.py", line 12, in <module>
              import salt.loader.context
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/loader/__init__.py", line 23, in <module>
              import salt.utils.event
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/utils/event.py", line 67, in <module>
              import salt.ext.tornado.iostream
            File "/var/tmp/.root_dd8a91_salt/pyall/salt/ext/tornado/iostream.py", line 42, in <module>
              import urllib3.util.ssl_match_hostname
          ModuleNotFoundError: No module named 'urllib3.util.ssl_match_hostname'

[unman]

I'm afraid this report is not particularly helpful, as many people are updating Debian templates without any such issue. We need to know what is specific to *this* set up that results in the error. The "Steps to reproduce " section is missing information about which Debian-11 template (full or minimal) is used, and what has been done to it. Does the error appear for *all* (and only) Debian-11 templates? Are there Debian templates which update correctly? Also information re management_dispvm used (check qvm-prefs) and what template is used by that qube. Also, have dom0 and other templates been updated?

[laoclos]

I am getting exact the same errors using the Qubes built-in update tools also for the default Whonix-16 (which is also based on Debian?!). The debian is a standalone clone of the default debian-11 template with not adjustments. Thus, no special adjustments on both systems. On another system running Qubes all updates were successful three or four days ago, without any issues.

[Minimalist73]

Does the error appear for all (and only) Debian-11 templates? Are there Debian templates which update correctly? Also information re management_dispvm used (check qvm-prefs) and what template is used by that qube. Also, have dom0 and other templates been updated?

I tested both debian-11 and debian-12, only debian-11 seems impacted (and other templates like whonix-16 too).
All templates are fully updated, that's when it started to do this.
The management qube use fedora-38 (fedora is not impacted by this).

This started yesterday after a salt update was installed on all my fedora-38 based templates (version 3006.2)
Debian 12 use a different salt version than debian-11 (3006.1-2 (12) / 3002.6 (11)), it can be the reason why it's not impacted.

[Minimalist73]

I tested a few more things with the management qube:

• Based on template debian-11:
  • Running salt version 3002.6+dfsg1-4+deb11u1
  • Working fine for debian-11 templates and no errors anywhere
• Based on template debian-12:
  • Running salt version 3006.1-2+deb12u1
  • Working fine for debian-11 templates and no errors anywhere
• Based on template fedora-38:
  • Running salt version 3006.2-1
  • Debian-11 show errors while debian-12 is working fine (difference in salt version between the template can be the reason)

Downgrading the salt version to 3006.1-2 on the fedora-38 template is fixing my issue for debian-11.

It means that the new salt version from the fedora-38 template (3006.2-1) is the reason why debian-11 based templates are showing this module error.

[unman]

@Minimalist Thank you for you detailed and comprehensive work. It is extremely helpful, and I wish other users were as capable.

[Szewcson]

Have the same issue also for some minimal fedora 38 templates (sys-net template and sys-vpn template while sys-usb template is unafected) and focal template. It starts after update of fedora 38 based management qube.