[BUG] 3006.5 Salt-minion service not responding and leaving defunct process

[djmmatracki]

Description
We have a scheduler job that executes a formula with large amount of states. After update to 3006.5 we noticed that minions stop responding after some time. The test we did was to execute a file with large amount of states (example in setup) using the scheduler. After running the state minions stop responding and they leave defunct process. We tested this behavior on a vanilla (default) salt-master and salt-minion setup.

Setup

/srv/salt/test_state.sls

{% for i in range(1, 1024) %}
echo_random_string_{{ i }}:
  cmd.run:
    - name: echo "This is a random string number {{ i }}"
{% endfor %}

Command for creating the job:

salt 'minion1' schedule.add test_job function='state.sls' job_args="['test_state']" seconds=300

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service - Linode
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior

After setting up the job and running following command:

salt 'minion1' schedule.run_job test_job

The minion stops responding leaving defunct process.

0:07      \_ [/opt/saltstack/] <defunct>

Expected behavior
The salt-minion should not leave a defunct process and should respond to salt commands.

Versions Report

salt --versions-report

Salt Version:
          Salt: 3006.5

Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: 1.10.6
         gitdb: 4.0.11
     gitpython: 3.1.40
        Jinja2: 3.1.2
       libgit2: 1.3.0
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: 2.6.1
  pycryptodome: 3.9.8
        pygit2: 1.7.0
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: 5.0.1
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 20.04.5 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-155-generic
        system: Linux
       version: Ubuntu 20.04.5 focal

[dwoz]

@djmmatracki Can you provide us some minion logs?

[djmmatracki]

Here are the salt-minion logs:

minion.txt

These logs show the process of:

• Running the scheduler with schedule.run_job (The job definition is specified in the issue description)
• Checking that the salt-minion process is defunct (Here we realize that the minion is not responding to commands)
• Killing the salt-minion process with kill command
• Starting the salt-minion service again
• Second test again

[viorelfoamete]

I encountered the same issue on 3007.1 and also after downgrading to 3006.9 (both masters and minions).
Minions that are running on the same server as masters are dying after some time. After some digging I found that they are dying every time after the state.highstate schedule job is running on them. It's strange that when the highstate was executed manually there was no issue.
Highstate was set to every 120 mins, moved it to every 5 mins and they kept dying. Killing the minion processes and restarting the minion services would do it for another 5 mins.

As a workaround I set return_job: false on the highstate scheduled job

schedule:
  schedule-highstate:
    enabled: true
    function: state.highstate
    jid_include: true
    kwargs: {queue: true}
    maxrunning: 1
    minutes: 5
    name: schedule-highstate
    return_job: false
    run: true
    splay: {end: 59, start: 0}

The affected minions are running ~650 states during highstate while the ones not affected around ~580.
Same resources are assigned to all minions.

[dmurphy18]

@djmmatracki Having problems duplicating this with the latest 3006 code which should be released soon as 3006.10

Will let this run for a while to see if time will surface the issue, running with seconds=60 to try speeding things along.
Just want to give you a heads up.

salt-call --local test.versions

```yaml local: Salt Version: Salt: 3006.9+777.g948888f531

Python Version:
        Python: 3.10.16 (main, Mar  6 2025, 02:23:15) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.18.1
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 24.04.2 noble
        locale: utf-8
       machine: x86_64
       release: 6.11.0-19-generic
        system: Linux
       version: Ubuntu 24.04.2 noble

</details>

[dmurphy18]

@djmmatracki Have not been able to duplicate this issue, wondering if there is anything else I can try to duplicate it, but there has been a lot of water under the bridge since 3006.5, and the latest 3006.9+plus has a lot fo changes and fixes. Given that the next Salt 3006.10 release is very soon, perhaps it would be best to wait for that release and see if the problem still exists for you.

[dmurphy18]

@djmmatracki Can you retry with the new Salt 3006.10 release from last week, a number of fixes were made about dropped processes.

[dmatrack]

Hi @dmurphy18 , we will retry with the new version. I will respond once we get the results :)