[BUG] salt-master memory leak

[raczkow]

Description
After upgrading from 3006.9 to 3007.1 of my salt-master, I've noticed memory leaks of master daemon which kills my server. There are about 250 minions connected. salt-master works about one day and it gets killed by oom-killer.

[~]# systemctl status salt-master.service 
● salt-master.service - The Salt Master Server
     Loaded: loaded (/lib/systemd/system/salt-master.service; enabled; vendor preset: enabled)
     Active: failed (Result: oom-kill) since Sat 2024-11-09 07:26:06 CET; 7h ago
       Docs: man:salt-master(1)
             file:///usr/share/doc/salt/html/contents.html
             https://docs.saltproject.io/en/latest/contents.html
    Process: 411021 ExecStart=/usr/bin/salt-master (code=exited, status=0/SUCCESS)
   Main PID: 411021 (code=exited, status=0/SUCCESS)
        CPU: 25min 23.240s

Setup
My system runs on LXC container in Proxmox 8.

salt-master -V
Salt Version:
          Salt: 3007.1
 
Python Version:
        Python: 3.10.14 (main, Apr  3 2024, 21:30:09) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: 18.8.0
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.16.0
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4
 
Salt Package Information:
  Package Type: onedir
 
System Versions:
          dist: debian 11.11 bullseye
        locale: utf-8
       machine: x86_64
       release: 6.8.12-3-pve
        system: Linux
       version: Debian GNU/Linux 11.11 bullseye

Steps to Reproduce the behavior
Run:

systemctl start salt-master

and wait a couple of hours.

Expected behavior
Memory usage is constant.

Screenshots
Here's the screenshot of memory usage of my system. Those peaks appeared after upgrade to 3007.1

[dwoz]

@raczkow Do you know which master process is leaking memory?

[raczkow]

PubServerChannel._publish_daemon

[raczkow]

This time it looks a little bit different:

[dwoz]

@raczkow is this still a problem?

[raczkow]

No, it works like a charm since may/june of 2025.

EDIT: I suppose, it was fixed in 3007.2 which I installed on May 18th.