[Bug]: 3006.24 regression: AttributeError: 'NoneType' object has no attribute 'set_result' in transport/ipc.py

[stecol11]

What happened?

Possibly related to 68992

Description of Bug:
After upgrading from 3006.23 to 3006.24 (Sulfur), salt-call consistently throws an AttributeError during the IPC handshake, even though the command itself (e.g., test.ping) returns the correct value.

Traceback:
[ERROR ] Future <salt.ext.tornado.concurrent.Future object at 0x7f35c5fde830> exception was never retrieved: Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/ipc.py", line 333, in _connect
self._connecting_future.set_result(True)
AttributeError: 'NoneType' object has no attribute 'set_result'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1070, in run
yielded = self.gen.send(value)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/ipc.py", line 343, in _connect
self._connecting_future.set_exception(e)
AttributeError: 'NoneType' object has no attribute 'set_exception'

Environment:

Salt Version: 3006.24

Python Version: 3.10.x

OS: Debian (Standard Saltstack Repo)

Notes: This appears to be a race condition in the IPC transport layer. Using salt-call --local bypasses the issue, confirming it is isolated to the Minion-to-CLI communication.

Steps to Reproduce:

Upgrade to 3006.24.

Run salt-call test.ping.

Observe the traceback before the True response.

Rolling back to 3006.23 resolved the issue for our entire fleet. This confirms it’s a regression and not a local environment failure.

Type of salt install

Official deb

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

debian-12

salt --versions-report output

Salt Version:
          Salt: 3006.24
 
Python Version:
        Python: 3.10.19 (main, Feb 17 2026, 23:32:03) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 2.0.0
      cherrypy: 18.10.0
  cryptography: 46.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.22.4
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: debian 12 bookworm
        locale: utf-8
       machine: x86_64
       release: 6.1.0-44-amd64
        system: Linux
       version: Debian GNU/Linux 12 bookworm

[lcsiki]

Same exact error happens on CL8 too.

Salt Version:
          Salt: 3006.24

Python Version:
        Python: 3.10.20 (main, Apr 22 2026, 05:54:37) [GCC 11.2.0]

Dependency Versions:
          cffi: 2.0.0
      cherrypy: unknown
  cryptography: 46.0.7
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: 3.1.46
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.22.8
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.5
           ZMQ: 4.3.4

System Versions:
          dist: cloudlinux 8.10 Vladimir Aksyonov
        locale: utf-8
       machine: x86_64
       release: 4.18.0-553.69.1.lve.el8.x86_64
        system: Linux
       version: CloudLinux 8.10 Vladimir Aksyono

[m0duspwnens]

Same error on Oracle Linux 9

Salt Version:
          Salt: 3006.24
 
Python Version:
        Python: 3.10.20 (main, Apr 22 2026, 05:54:37) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 2.0.0
      cherrypy: unknown
  cryptography: 46.0.7
      dateutil: 2.8.2
     docker-py: 7.1.0
         gitdb: 4.0.12
     gitpython: 3.1.46
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.22.8
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.5
           ZMQ: 4.3.4
 
System Versions:
          dist: oracle 9.7 
        locale: utf-8
       machine: x86_64
       release: 5.15.0-319.201.4.2.el9uek.x86_64
        system: Linux
       version: Oracle Linux Server 9.7

[OrangeDog]

And, as probably expected, same on Ubuntu 24.04.

Salt Version:
          Salt: 3006.24

Python Version:
        Python: 3.10.20 (main, Apr 22 2026, 05:54:37) [GCC 11.2.0]

Dependency Versions:
          cffi: 2.0.0
      cherrypy: unknown
  cryptography: 46.0.7
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.22.8
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.5
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 24.04.4 noble
        locale: utf-8
       machine: x86_64
       release: 6.8.0-110-generic
        system: Linux
       version: Ubuntu 24.04.4 noble

[racooper]

I can confirm the same on RHEL 9; I had only upgraded my master and its own minion so far, and started seeing this error on a nightly cron job that does a state.sls call. Cron is calling it via salt CLI, but I reproduced it using salt-call (without --local) too.

Salt Version:
          Salt: 3006.24
 
Python Version:
        Python: 3.10.20 (main, Apr 22 2026, 05:54:37) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 2.0.0
      cherrypy: unknown
  cryptography: 46.0.7
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: 3.1.46
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.22.8
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.5
           ZMQ: 4.3.4
 
System Versions:
          dist: rhel 9.7 Plow
        locale: utf-8
       machine: x86_64
       release: 5.14.0-611.49.1.el9_7.x86_64
        system: Linux
       version: Red Hat Enterprise Linux 9.7 Plow

[anandb-ripencc]

Does anyone know what the effect of this bug is? I am also seeing the error, but highstate appears to be running just fine. I want to understand whether this is just causing noise in the logs, or an actual problem.

[anandb-ripencc]

At least one of the effects of this bug appears to be some kind of memory leak. We upgraded one salt master to 3006.24, and this morning, the salt-master process was consuming more than 14G, and we got warnings from our monitoring. I rolled it back to 3006.23.

[OrangeDog]

@anandb-ripencc this issue can be seen only updating the minion, so your new master leak may be something else.

[anandb-ripencc]

@OrangeDog oh, sorry, I was not clear enough. The master is also a minion, so the minion log was showing the error. Additionally, there was another remote minion, running 3006.24, also talking to this master. I'm attaching a graph of the memory use of the server, and you can clearly see when the leak began (after upgrading to 3006.24) and when it stabilised (after downgrading to 3006.23). Perhaps the minion-side bug is caused the master's memory usage to grow.

[OrangeDog]

@anandb-ripencc yes, that clearly shows this release has a new memory leak in the master, but there is nothing connecting it to this issue. You should make a separate one.

[anderssynstad]

Seeing this on Debian 13 (Trixie) as well. (Just to add to the OS flavour mix here).