[BUG] Possible concurrency issue with reactor/orchestrate running on master only

[dpizzle]

Description
I have a salt master which is configured with 3 reactor/orchestrate pairs. The master listens to messages from network devices via napalm-logs, then performs 3 tasks -

1. updates a mariadb with the event details

2. updates the db with the host details

3. runs a python script which connects to the network device and collects further information to add to the db.

As the number of messages has increased, I'm noticing that more of the jobs are failing with either of these 2 responses. The errors are sporadic and not linked to specific network devices.

Can state.orchestrate only run one job at a time, so if a second state.orchestrate is called from the reactor, whilst the first is running, the second job will fail?

"The function \"state.orchestrate\" is running as PID 6668 and was started at 2020, Jun 10 19:11:53.798538 with jid 20200610191153798538"

"Rendering SLS 'base:orchestrate.register_host_in_db' failed: Jinja variable 'salt.utils.context.NamespacedDictWrapper object' has no attribute 'data'"

Setup

/etc/salt # more master.d/reactors.conf
reactor:
  - 'napalm/syslog/*/*/*':
    - salt://reactor/register_host_in_db.sls
    - salt://reactor/register_event_in_db.sls

  - 'napalm/syslog/junos/PAM_AUTH/*':
    - salt://reactor/interrogate_node.sls

/etc/salt # more reactor/register_host_in_db.sls

register host in db:
  runner.state.orchestrate:
    - args:
      - mods: orchestrate.register_host_in_db
      - pillar:
          data: {{ data|json }}

/etc/salt # more orchestrate/register_host_in_db.sls
  register host in db - {{ hostname }}:
    mysql_query.run:
      - database: db
      - query: |
          INSERT ...

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
           Salt: 2019.2.2

Dependency Versions:
           cffi: 1.14.0
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 1.0.0
   mysql-python: Not Installed
      pycparser: 2.20
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.9 (default, Oct 17 2019, 11:10:22)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 19.0.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.2

System Versions:
           dist:
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1062.12.1.el7.x86_64
         system: Linux
        version: Not Installed

Additional Information

I've upgraded to 3000.3 and still experience the same behaviour.

[dpizzle]

To remove the 3 tasks from the troubleshooting, I've replaced with one tasks that writes to the log file and still have the same issue.

/etc/salt # more master.d/reactors.conf
reactor:
  - 'napalm/syslog/*/*/*':
    - salt://reactor/debugger.sls

/etc/salt # more reactor/debugger.sls
debug:
  runner.state.orchestrate:
    - args:
      - mods: orchestrate.debugger
      - pillar:
          data: {{ data|json }}

/etc/salt # more orchestrate/debugger.sls
{# log host and pillar data to /var/log/salt/master #}
{% do salt.log.warning("orch.debugger: host:" + pillar['data']['host']|string|lower + " data:" + pillar['data']|string) %}

fix empty state - debugger:
  cmd.run:
    - name: /bin/true
    - unless: /bin/true

[xeacott]

Interesting, thanks for the report @dpizzle unfortunately I don't have much experience with reactors so I'll ask if anyone on @saltstack/team-core is able to assist here. 😄