[BUG] Reactor executes action multiple times

[supfors]

Description
We use a state that sends an event to the bus. Reactor is then to run a state (local.state.apply) to the minion. In salt 3005 I've noticed that the local.state.apply is executed multiple times on the same minion. When the event is sent by 1 or 2 minions at the same time, the reactor state is always executed twice. When using more minions, i.e. 10, the state is executed twice by some of the minions, but not all.

Setup
/etc/salt/master:

reactor:
  - 'debug/test':
    - salt://debug/reactor.sls

/srv/salt/debug/send.sls:

{{ sls }} Sent event to salt:
  event.send:
    - name: debug/test

/srv/salt/debug/reactor.sls:

execute_state:
  local.state.apply:
    - tgt: {{ data.id }}
    - arg:
      - debug.action

/srv/salt/debug/action.sls:

/tmp/test.txt:
  file.managed:
    - contents: {{ grains.id }} -- {{ None|strftime("%A %B %d %Y %H:%M:%S") }}

salt -L 'domi1,domi2' state.apply debug.send

salt-run jobs.list_jobs --out=txt | sort -h |grep debug

20220908083857000400: {'Function': 'state.apply', 'Arguments': ['debug.send'], 'Target': ['domi1', 'domi2'], 'Target-type': 'list', 'User': 'sudo_root', 'StartTime': '2022, Sep 08 08:38:57.000400'}
20220908083859330938: {'Function': 'state.apply', 'Arguments': ['debug.action'], 'Target': 'domi2', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2022, Sep 08 08:38:59.330938'}
20220908083859386338: {'Function': 'state.apply', 'Arguments': ['debug.action'], 'Target': 'domi1', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2022, Sep 08 08:38:59.386338'}
20220908083906102574: {'Function': 'state.apply', 'Arguments': ['debug.action'], 'Target': 'domi2', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2022, Sep 08 08:39:06.102574'}
20220908083906388266: {'Function': 'state.apply', 'Arguments': ['debug.action'], 'Target': 'domi1', 'Target-type': 'glob', 'User': 'root', 'StartTime': '2022, Sep 08 08:39:06.388266'}

salt-run jobs.lookup_jid --out=highstate --state-output=changes 20220908083859386338

domi1:
----------
          ID: /tmp/test.txt
    Function: file.managed
      Result: True
     Comment: File /tmp/test.txt updated
     Started: 08:38:59.758486
    Duration: 13.878 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1 +1 @@
                  -domi1 -- Thursday September 08 2022 08:26:42
                  +domi1 -- Thursday September 08 2022 08:38:59

Summary for domi1
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  13.878 ms

salt-run jobs.lookup_jid --out=highstate --state-output=changes 20220908083906388266

domi1:
----------
          ID: /tmp/test.txt
    Function: file.managed
      Result: True
     Comment: File /tmp/test.txt updated
     Started: 08:39:06.729324
    Duration: 15.12 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1 +1 @@
                  -domi1 -- Thursday September 08 2022 08:38:59
                  +domi1 -- Thursday September 08 2022 08:39:06

Summary for domi1
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  15.120 ms

Expected behavior
I expect action.sls to be executed once per event sent by minions.

Versions Report

Salt Version:
          Salt: 3005
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: 5.0.3
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.0
       libgit2: 1.5.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.10.1
        Python: 3.9.13 (main, Aug 23 2022, 18:31:15)
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 22.04 jammy
        locale: utf-8
       machine: x86_64
       release: 5.15.0-47-generic
        system: Linux
       version: Ubuntu 22.04 jammy

Additional context

• I did not see this behaviour in salt 3004.2 (both the master and minions on the same version)

[supfors]

Just tested this with 3005.1 (both master and minion), issue is still present.

[mastrong]

Edit: I happened to be using the only minion that I have setup for multi-master (and had forgotten). I am not seeing duplicate reactions now that I have removed one.

[tuxthepenguin84]

Can confirm i'm seeing the same issue with reactors firing twice on the salt-master on 3005.1. I have my minions (3005.1) configured for multi-master, and if I remove the multi-master config and got back to a single master the reactors only fire once (which is expected).

Edit:
I did some more testing and by only reverting the minion to 3004.2 (leave the master on 3005.1) the issue goes away.

[a-schuurman]

I can confirm the same issue, in my case all reactor calls are executed twice. Most of our infrastructure is idempotent, so it won't hurt the business. Unfortunately reactor actions take up twice as much time and resources, non-idempotent operations create additional artefacts because of this behaviour.

[tuxthepenguin84]

For those of you that are looking for an alternative to Salt's builtin multi-master, keepalived is an alternative that does work. Once keepalived is configured properly and running I added this to my minion config:

master_tries: -1

ping_interval: 1

tcp_keepalive: True
tcp_keepalive_idle: 30
tcp_keepalive_cnt: 1
tcp_keepalive_intvl: 30