[BUG] salt-ssh render pillar twice (once as master, once as minion)

[s-at-ik]

Description

Pillar rendering (tested with python she-bang and ext_pillar) happens twice when using salt-ssh: once in the master context, once in the minion context.

Setup

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD

Using Debian bullseye vms (official cloud image) running on a private openstack cloud. Using official salt latest repo. Nothing beside salt installed. Everything up-to-date.

Steps to Reproduce the behavior

I setup a configuration key on the master and then read it in a python pillar (this is obviously totally useless but the simplest way I found to reproduce/show the problem).

root@salt-playground-master:~# cat /etc/salt/master.d/test.conf 
foo: bar

root@salt-playground-master:~# cat /srv/pillar/top.sls 
base:
  "*":
    - test
root@salt-playground-master:~# cat /srv/pillar/test.sls 
#!py

import logging

log = logging.getLogger(__name__)

def run():
    log.warning("test pillar called")
    return {"foo": __opts__["foo"]}

root@salt-playground-master:~# cat /srv/salt/top.sls 
base:
  "*":
    - test
root@salt-playground-master:~# cat /srv/salt/test.sls 
test:
  test.nop:
    - foo: {{ salt["pillar.get"]("foo") }}

When applying or displaying the state using master + minion, everything works as expected:

root@salt-playground-master:~# salt salt-playground-minion state.show_sls test
salt-playground-minion:
    ----------
    test:
        ----------
        __env__:
            base
        __sls__:
            test
        test:
            |_
              ----------
              foo:
                  bar
            - nop
            |_
              ----------
              order:
                  10000
root@salt-playground-master:~# cat /var/log/salt/master 
2022-06-28 11:44:22,804 [test             :8   ][WARNING ][4809] test pillar called

But when using salt-ssh:

root@salt-playground-master:~# salt-ssh salt-playground-ssh state.show_sls test
[WARNING ] test pillar called
[WARNING ] test pillar called
[CRITICAL] Rendering SLS 'test' failed, render error:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 699, in py
    data = mod.run()
  File "/srv/pillar/test.sls", line 4, in run
    return {"foo": __opts__["foo"]}
KeyError: 'foo'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/pillar/__init__.py", line 887, in render_pstate
    state = compile_template(
  File "/usr/lib/python3/dist-packages/salt/template.py", line 99, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/renderers/py.py", line 147, in render
    raise SaltRenderError(
salt.exceptions.SaltRenderError: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 699, in py
    data = mod.run()
  File "/srv/pillar/test.sls", line 4, in run
    return {"foo": __opts__["foo"]}
KeyError: 'foo'

[CRITICAL] Pillar render error: Rendering SLS 'test' failed. Please see master log for details.
salt-playground-ssh:
    ----------
    test:
        ----------
        __env__:
            base
        __sls__:
            test
        test:
            |_
              ----------
              foo:
                  bar
            - nop
            |_
              ----------
              order:
                  10000

The pillar appear to be rendered twice (see warning at the start) and the second time does not have access to the master's configuration. The first render appears to work as intended as the value is present in the pillar.

Expected behavior

State should behave the same when using salt-ssh and render the pillar only once, on the master.

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

root@salt-playground-master:~# salt --versions-report
Salt Version:
          Salt: 3004.2
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.5
     gitpython: 3.1.14
        Jinja2: 2.11.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.9.7
        pygit2: Not Installed
        Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 20.0.0
         smmap: 4.0.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-15-cloud-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye

salt-ssh --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

root@salt-playground-master:~# salt-ssh --versions-report
Salt Version:
          Salt: 3004.2
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.5
     gitpython: 3.1.14
        Jinja2: 2.11.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.9.7
        pygit2: Not Installed
        Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 20.0.0
         smmap: 4.0.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-15-cloud-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[Stogas]

We're also affected by this issue.

Additionally, salt-ssh did not respect our master's conf option git_pillar_provider: gitpython, reverting to the default pygit2 preference for the 2nd pillar/state render (until we force uninstalled pygit2 from the master's OS).

Versions Report

salt --versions-report

# salt --versions-report
Salt Version:
          Salt: 3004.1

Dependency Versions:
          cffi: 1.15.0
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: 4.1.0
         gitdb: 4.0.5
     gitpython: 3.1.24
        Jinja2: 2.11.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.7
        pygit2: Not Installed
        Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 20.0.0
         smmap: 4.0.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-13-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye

salt-ssh --versions-report

# salt-ssh --versions-report
Salt Version:
          Salt: 3004.1

Dependency Versions:
          cffi: 1.15.0
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: 4.1.0
         gitdb: 4.0.5
     gitpython: 3.1.24
        Jinja2: 2.11.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.7
        pygit2: Not Installed
        Python: 3.9.2 (default, Feb 28 2021, 17:03:44)
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 20.0.0
         smmap: 4.0.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-13-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye