[BUG] SSH orch/saltutil.cmd with ssh=True breaks gpg renderer and mine/publish wrappers

[lkubb]

Description
When using the orchestrate runner with salt.state and ssh: true, loaded mods for the SSH minion have slightly different __opts__ than when executing salt-ssh directly. This breaks the gpg renderer and the roster with nested calls.

This has the similar symptoms as #60002 (fixed in #65484), but a slightly different cause.

Setup

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• onedir packaging
• used bootstrap to install

I'm running with the above-mentioned patch, otherwise the GPG renderer would be broken with salt-ssh as well.

Steps to Reproduce the behavior

# /srv/salt/orch/ssh_hs.sls
Run a highstate on an SSH minion:
  salt.state:
    - tgt: myminion
    - highstate: true
    - ssh: true

• Have a GPG-encrypted pillar value for an SSH minion present in the roster
• Either run a dev version with the above patch or apply it (otherwise salt-ssh pillar rendering will also be broken)
• Try salt-ssh myminion state.highstate and see that it works
• Try salt-run state.orch orch.ssh_hs and see that it fails (or salt-call saltutil.cmd myminion state.highstate ssh=true)

Expected behavior
Work as usual

Screenshots

[WARNING ] Could not decrypt cipher '-----BEGIN PGP MESSAGE-----\n\n[...]\n-----END PGP MESSAGE-----', received: "gpg: keyblock resource '/var/tmp/.root_abcdef_salt/gpgkeys/pubring.kbx': No such file or directory\n[GNUPG:] ERROR add_keyblock_resource 33587281\n[GNUPG:] ENC_TO 01234567890ABCDEF 18 0\n[GNUPG:] ERROR keydb_search 33554445\ngpg: encrypted with ECDH key, ID 01234567890ABCDEF\ngpg: failed to create temporary file '/var/tmp/.root_abcdef_salt/gpgkeys/.#lk0x0000556cb744f110.salt1.1336331': No such file or directory\ngpg: can't connect to the gpg-agent: No such file or directory\n[GNUPG:] ERROR keydb_search 33554445\n[GNUPG:] NO_SECKEY 01234567890ABCDEF\ngpg: public key decryption failed: No secret key\n[GNUPG:] ERROR pkdecrypt_failed 33554449\n[GNUPG:] BEGIN_DECRYPTION\n[GNUPG:] DECRYPTION_FAILED\ngpg: decryption failed: No secret key\n[GNUPG:] END_DECRYPTION\n"
/opt/saltstack/salt/lib/python3.10/site-packages/salt/renderers/gpg.py:443: DeprecationWarning: After the Chlorine release of Salt, gpg_decrypt_must_succeed will default to True.
  salt.utils.versions.warn_until(
[WARNING ] Could not decrypt cipher '-----BEGIN PGP MESSAGE-----\n\n[...]\n-----END PGP MESSAGE-----', received: "gpg: keyblock resource '/var/tmp/.root_abcdef_salt/gpgkeys/pubring.kbx': No such file or directory\n[GNUPG:] ERROR add_keyblock_resource 33587281\n[GNUPG:] ENC_TO 01234567890ABCDEF 18 0\n[GNUPG:] ERROR keydb_search 33554445\ngpg: encrypted with ECDH key, ID 01234567890ABCDEF\ngpg: failed to create temporary file '/var/tmp/.root_abcdef_salt/gpgkeys/.#lk0x0000558a501e9110.salt1.1336332': No such file or directory\ngpg: can't connect to the gpg-agent: No such file or directory\n[GNUPG:] ERROR keydb_search 33554445\n[GNUPG:] NO_SECKEY 01234567890ABCDEF\ngpg: public key decryption failed: No secret key\n[GNUPG:] ERROR pkdecrypt_failed 33554449\n[GNUPG:] BEGIN_DECRYPTION\n[GNUPG:] DECRYPTION_FAILED\ngpg: decryption failed: No secret key\n[GNUPG:] END_DECRYPTION\n"
[...]

Patching the above to look in __opts["__master_opts__"] if set, another error transpires (I'm using the publish and mine wrappers during state rendering):

Can't access roster for backend flat: Roster file "/var/tmp/.root_abcdef_salt/roster" not found

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3006.4

Python Version:
        Python: 3.10.13 (main, Oct  4 2023, 21:54:22) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.13.12
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: rocky 9.2 Blue Onyx
        locale: utf-8
       machine: x86_64
       release: 5.14.0-284.18.1.el9_2.x86_64
        system: Linux
       version: Rocky Linux 9.2 Blue Onyx

Additional context
After a lot of debugging and suspecting the loader, I found that the issue is simply that instantiating salt.client.ssh.client.SSHClient does not populate config_dir in the SSH master opts (which is a difference to the CLI variant). This results in

a) config_dir being set wrong during pillar rendering (it's taken from the remote minion opts)
b) config_dir missing from __opts__["__master_opts__"].

(a) breaks the GPG renderer during pillar rendering.
(b) breaks the workaround inside salt.roster.Roster.get_roster_file for nested SSH calls.

[lkubb]

Fixed by #65673