[Bug]: loader race condition causes OS-specific virtual modules (e.g. postgres) to randomly fail

[LorenzoDeBie]

What happened?

Description

After upgrading to Salt 3006.26, we started seeing random state failures indicating that the postgres module is unavailable.

This is caused by a race condition in the lazy loader introduced in commit b45b721. When a module with an OS-specific implementation (like deb_postgres.py) evaluates its __virtual__ function and returns False (because it's not running on Debian), the loader intercepts this in salt/loader/lazy.py and overwrites the module_name with the __virtualname__ (which is "postgres").

This caches the failure as self.missing_modules['postgres'] rather than self.missing_modules['deb_postgres']. Because of non-deterministic file loader ordering, if deb_postgres.py is evaluated before the native postgres.py, the postgres virtualname gets poisoned in the missing cache. When the loader subsequently tries to load the valid postgres.py module, it skips it entirely because it's marked as missing (as seen in

salt/salt/loader/lazy.py

Line 1168 in f2c62b1

if mod_name in self.missing_modules or key in self._dict:

or

salt/salt/loader/lazy.py

Line 1217 in f2c62b1

if name in self.loaded_files or name in self.missing_modules:

)

This causes states to fail with a red herring error message:

----------
          ID: <redacted>
    Function: postgres_user.present
        Name: <redacted>
      Result: False
     Comment: State 'postgres_user.present' was not found in SLS '<redacted>'
              Reason: 'postgres_user' __virtual__ returned False: Unable to load postgres module.  Make sure `postgres.bins_dir` is set.
     Changes:
----------
          ID: <redacted>
    Function: postgres_user.present
        Name: <redacted>
      Result: False
     Comment: State 'postgres_user.present' was not found in SLS '<redacted>'
              Reason: 'postgres_user' __virtual__ returned False: Unable to load postgres module.  Make sure `postgres.bins_dir` is set.
     Changes:
----------
          ID: <redacted>
    Function: postgres_database.present
        Name: <redacted>
      Result: False
     Comment: State 'postgres_database.present' was not found in SLS '<redacted>'
              Reason: 'postgres_database' __virtual__ returned False: Unable to load postgres module.  Make sure `postgres.bins_dir` is set.
     Changes:

The states failing has nothing to do with postgres.bins_dir.

Setup

• on-prem machine
• VM (Proxmox)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior

1. Run Salt 3006.26 on a non-Debian OS (e.g., Rocky Linux 10.2 or RHEL).
2. Execute a state or execution module that relies on the postgres module.
3. Because Python's directory traversal is non-deterministic, if deb_postgres.py is parsed by the loader before postgres.py, the deb_postgres virtual check fails, poisons the postgres virtualname in missing_modules, and causes the run to fail saying the postgres module isn't available.

Expected behavior

Other modules claiming the same __virtualname__ (postgres) can still successfully load.

Root Cause

In salt/loader/lazy.py, inside _process_virtual during commit b45b721

                     # If the module explicitly declares __virtualname__, report
                     # the failure under that name so the caller can detect
                     # collisions with other modules claiming the same name.
                    if (
                        hasattr(mod, "__virtualname__")
                        and isinstance(virtualname, str)
                        and virtualname
                     ):
                        module_name = virtualname

                    return (False, module_name, error_reason, virtual_aliases)

was added. I debugged two servers, one running 3006.25 and one running 3006.26. I found this piece of code was the difference between the two and caused the postgres states to fail.
When virtual is False, overriding module_name with virtualname directly triggers the bug. Removing this entire if block resolves the issue and allows the loader to correctly register deb_postgres in missing_modules, leaving postgres clean for the actual module.

Type of salt install

Official rpm

Major version

3006.x

What supported OS are you seeing the problem on? Can select multiple. (If bug appears on an unsupported OS, please open a GitHub Discussion instead)

rockylinux-10

salt --versions-report output

Salt Version:
          Salt: 3006.26

Python Version:
        Python: 3.10.20 (main, Jun 11 2026, 02:19:33) [GCC 11.2.0]

Dependency Versions:
          cffi: 2.0.0
      cherrypy: 18.10.0
  cryptography: 47.0.0
      dateutil: 2.9.0.post0
     docker-py: Not Installed
         gitdb: 4.0.12
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.1.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 3.00
      pycrypto: Not Installed
  pycryptodome: 3.23.0
        pygit2: Not Installed
  python-gnupg: 0.5.6
        PyYAML: 6.0.3
         PyZMQ: 27.1.0
        relenv: 0.22.14
         smmap: 5.0.2
       timelib: 0.3.0
       Tornado: 6.5.5
           ZMQ: 4.3.5

System Versions:
          dist: rocky 10.1 Red Quartz
        locale: utf-8
       machine: x86_64
       release: 6.12.0-124.8.1.el10_1.x86_64
        system: Linux
       version: Rocky Linux 10.1 Red Quartz