[BUG] salt-ssh fails with TypeError: '>' not supported between instances of 'NoneType' and 'int'

[marmarek]

Description

Sometimes (haven't figured out when) salt-ssh fails with TypeError: '>' not supported between instances of 'NoneType' and 'int'. This seems to be a regression caused by #64515.

Setup
The state files used are at https://github.com/QubesOS/openqa-tests-qubesos/tree/main/extra-files (system-tests and update subdirs, all enabled). But I do not believe the issue is specific to some states.

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify) - this is Qubes VM, using integration at https://github.com/QubesOS/qubes-mgmt-salt. For all intents and purposes, it behaves like SSH into a Xen VM.
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior

Call salt-ssh remote-host state.highstate.

  [ERROR   ] An un-handled exception was caught by Salt's global exception handler:
  TypeError: '>' not supported between instances of 'NoneType' and 'int'
  Traceback (most recent call last):
    File "/usr/bin/salt-ssh", line 8, in <module>
      sys.exit(salt_ssh())
               ^^^^^^^^^^
    File "/usr/lib/python3.11/site-packages/salt/scripts.py", line 471, in salt_ssh
      client.run()
    File "/usr/lib/python3.11/site-packages/salt/cli/ssh.py", line 20, in run
      ssh.run()
    File "/usr/lib/python3.11/site-packages/salt/client/ssh/__init__.py", line 810, in run
      final_exit = max(final_exit, retcode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  TypeError: '>' not supported between instances of 'NoneType' and 'int'
  Traceback (most recent call last):
    File "/usr/bin/salt-ssh", line 8, in <module>
      sys.exit(salt_ssh())
               ^^^^^^^^^^
    File "/usr/lib/python3.11/site-packages/salt/scripts.py", line 471, in salt_ssh
      client.run()
    File "/usr/lib/python3.11/site-packages/salt/cli/ssh.py", line 20, in run
      ssh.run()
    File "/usr/lib/python3.11/site-packages/salt/client/ssh/__init__.py", line 810, in run
      final_exit = max(final_exit, retcode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  TypeError: '>' not supported between instances of 'NoneType' and 'int'

Expected behavior

No exception, instead a normal state output. I suspect some of the states (or the whole state.highstate) failed here, but unfortunately I have no further logs, due to this exception.

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3006.1
 
Python Version:
        Python: 3.11.3 (main, May 24 2023, 00:00:00) [GCC 13.1.1 20230511 (Red Hat 13.1.1-2)]
 
Dependency Versions:
          cffi: 1.15.1
      cherrypy: Not Installed
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.0.3
       libgit2: Not Installed
  looseversion: 1.2.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.4
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.0
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.17.0
        pygit2: Not Installed
  python-gnupg: Not Installed
        PyYAML: 6.0
         PyZMQ: 24.0.1
        relenv: Not Installed
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: fedora 38 
        locale: utf-8
       machine: x86_64
       release: 6.3.9-1.qubes.fc37.x86_64
        system: Linux
       version: Fedora Linux 38 

This is 3006.1 on Fedora 38, custom package with #64515 backported.

Additional context
Add any other context about the problem here.

[lkubb]

This regression was also reproduced during an integration test of another PR: https://github.com/saltstack/salt/actions/runs/5409628303/jobs/9831926259?pr=64542

Since this report references a state.highstate, I assumed this was the result of the remote reporting a null retcode in

salt/salt/client/ssh/__init__.py

Lines 555 to 559 in c463c94

	try:
	# Ensure a reported local retcode is kept
	retcode = data["local"]["retcode"]
	except (KeyError, TypeError):
	pass

The logs of the failing integration test above suggest the None retcode actually comes from a non-wrapped call via salt.utils.vt.Terminal (for whatever reason):

10:06:24,168 [salt.utils.vt                                                                   :289 ][DEBUG   ][Process-1(553)] [SaltMaster(id='master')] VT: Salt-SSH SHIM Terminal Command executed. Logged to TRACE
10:06:27,887 [tests.support.case                                                              :289 ][DEBUG   ][MainProcess(31943)] run_script results for: salt-ssh  -W  -l error --ignore-host-keys --priv /tmp/stsuite/sshd/client_key --roster-file /tmp/stsuite/master/conf/roster  localhost file.file_exists /tmp/salt_test_file_pillar --out=json
return code: 1
stdout:


stderr:
None
10:06:27,888 [tests.support.case                                                              :289 ][DEBUG   ][MainProcess(31943)] Result of run_ssh for command ' -W  -l error --ignore-host-keys --priv /tmp/stsuite/sshd/client_key --roster-file /tmp/stsuite/master/conf/roster  localhost file.file_exists /tmp/salt_test_file_pillar --out=json {}': 
10:06:27,889 [tests.support.case                                                              :289 ][DEBUG   ][MainProcess(31943)] SSHCase run_function executed file.file_exists with arg ['/tmp/salt_test_file_pillar'] and kwargs {}
10:06:27,889 [tests.support.case                                                              :289 ][DEBUG   ][MainProcess(31943)] SSHCase JSON return: 

[dwoz]

Closing as fixed per #65067 (comment)