[BUG] salt-syndic memory leak

[terminalmage]

Description

Over time, the syndic process grows in memory until all memory is exhausted and the syndic gets OOM-killed.

This syndic processes a lot of events, forwarding anywhere from 150-600 at a time (discovered through adding some debug logging to salt.minion.SyndicManager._forward_events()).

I have used a few profilers to discover where memory is being allocated, and they all point to msgpack. Here's the relevant part of the summary from running the syndic using scalene:

                           Memory usage: ▁▂▂▃▃▃▃▃▃▄▄▄▅▅▅▆▆▆▆▇▇██████ (max: 15.046 GB, growth rate:  28%)

... snip...

           /usr/local/lib/python3.9/dist-packages/msgpack/__init__.py: % of time =   0.36% (8.694s) out of 41m:41.056s.
       ╷       ╷       ╷       ╷        ╷       ╷               ╷       ╷
       │Time   │–––––– │–––––– │Memory  │–––––– │–––––––––––    │Copy   │
  Line │Python │native │system │Python  │peak   │timeline/%     │(MB/s) │/usr/local/lib/python3.9/dist-packages/msgpack/__init__…
╺━━━━━━┿━━━━━━━┿━━━━━━━┿━━━━━━━┿━━━━━━━━┿━━━━━━━┿━━━━━━━━━━━━━━━┿━━━━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸

... snip ...

    30 │       │       │       │        │       │               │       │def packb(o, **kwargs):
    31 │       │       │       │        │       │               │       │    """
    32 │       │       │       │        │       │               │       │    Pack object `o` and return packed bytes
    33 │       │       │       │        │       │               │       │
    34 │       │       │       │        │       │               │       │    See :class:`Packer` for options.
    35 │       │       │       │        │       │               │       │    """
    36 │       │       │       │ 100%   │11.85G │▁▁▂▂▂▃▃▃▃  37% │     9 │    return Packer(**kwargs).pack(o)
    37 │       │       │       │        │       │               │       │

... snip ...

╶──────┼───────┼───────┼───────┼────────┼───────┼───────────────┼───────┼─────────────────────────────────────────────────────────╴
       │       │       │       │        │       │               │       │function summary for /usr/local/lib/python3.9/dist-pack…
    30 │       │       │       │ 100%   │11.85G │█████████  37% │     9 │AsyncReqMessageClient.packb
       ╵       ╵       ╵       ╵        ╵       ╵               ╵       ╵

Setup

There is nothing special about the syndic configuration. The syndic_master is configured with the IP of the master-of-masters, and order_masters is set to True. The master running on the syndic is using the default number of worker threads (5).

The box is running under KVM virtualization, with Salt installed via pip.

Versions Report

          Salt: 3006.5

Python Version:
        Python: 3.9.2 (default, Feb 28 2021, 17:03:44)

Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 24.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.20.0
        pygit2: Not Installed
  python-gnupg: Not Installed
        PyYAML: 5.3.1
         PyZMQ: 25.1.2
        relenv: Not Installed
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-28-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye