Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Events can grow stale when event_listen_queue is set. #53411

Open
cro opened this issue Jun 7, 2019 · 0 comments

Comments

@cro
Copy link
Member

commented Jun 7, 2019

Description of Issue

When event_listen_queue on the master is set to large values (to reduce load and increase performance on busy infrastructures), in some cases there can be a lull in event traffic that will cause events in the listen queue to get stale. This creates the impression that jobs are taking much longer to complete than they should.

Steps to Reproduce Issue

Set event_listen_queue in /etc/salt/master to a large value (2500 for example). Restart the master. Wait for minions to reconnect. Run some commands (test.ping is sufficient). Note that it can take a long time for results to come back and/or the CLI may report minions as being disconnected.

Versions Report

Salt Version:
           Salt: 2018.3.0-1228-g07a3112

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
        libnacl: 1.6.1
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.1
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15rc1 (default, Nov 12 2018, 14:31:15)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 18.0.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.9.125-linuxkit
         system: Linux
        version: Ubuntu 18.04 bionic
cro added a commit to cro/salt that referenced this issue Jun 7, 2019
@cro cro referenced this issue Jun 7, 2019
@Ch3LL Ch3LL added this to the Approved milestone Jun 11, 2019
cro added a commit to cro/salt that referenced this issue Jun 14, 2019
cro added a commit to cro/salt that referenced this issue Jun 14, 2019
This reverts commit eb3d8b8.
cro added a commit to cro/salt that referenced this issue Jun 18, 2019
cro added a commit to cro/salt that referenced this issue Jun 24, 2019
cro added a commit to cro/salt that referenced this issue Jun 24, 2019
cro added a commit that referenced this issue Jun 25, 2019
Forward port #53412 to 2019.2: Add event_listen_queue_max_seconds to fix #53411
cro added a commit that referenced this issue Jun 28, 2019
Forward port from 2018.3 Add event_listen_queue_max_seconds to fix #53411
sbrennan4 pushed a commit to bloomberg/salt that referenced this issue Aug 19, 2019
sbrennan4 pushed a commit to bloomberg/salt that referenced this issue Aug 19, 2019
Add event_listen_queue_max_seconds to fix saltstack#53411
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.