Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stale events 53411 #53412

Merged
merged 6 commits into from Jun 28, 2019

Conversation

@cro
Copy link
Member

commented Jun 7, 2019

What does this PR do?

Adds event_listen_queue_max_seconds to fix #53411

What issues does this PR fix or reference?

#53411

Previous Behavior

If event_listen_queue is set to large values or if it is set but the event bus is not busy enough, events can languish in the queue and cause pathological behavior (CLI reporting Minion did not return, job results taking too long to return.

New Behavior

The event_listen_queue will be flushed if the oldest event in the queue is older than event_listen_queue_max_seconds regardless of the number of events in the queue.

Commits signed with GPG?

Yes

@cro cro requested a review from saltstack/team-core as a code owner Jun 7, 2019
@cro cro changed the base branch from develop to 2018.3 Jun 7, 2019
@dwoz

This comment has been minimized.

Copy link
Contributor

commented Jun 7, 2019

Looks good but we are waiting on some tests for this change.

Copy link
Contributor

left a comment

The extraneous conditional
else:
too_long_in_queue = False

might be unneeded, since line 1272 resets to False, unless I missed the need for it, in which case let me know what I missed.
otherwise looks good,

@cro

This comment has been minimized.

Copy link
Member Author

commented Jun 18, 2019

After banging my head against this for some time I think we will need to resort to a manual test.

  1. The test suite currently starts a master but never restarts one so unless we make this config the default there's no way to alter the configuration
  2. There are no tests (that I can find) in the suite that verify functionality of non-default master config options.

If there is an approach I am unaware of I'm happy to explore it.

@waynew

This comment has been minimized.

Copy link
Contributor

commented Jun 18, 2019

Both of those points sound accurate to me.

I would love it if we had an automated suite that roughly does what we do with the current integration suite, but instead was more like just running a salt orchestration, and we verified that each of the state result was what we expected. Basically an automated manual test suite 😂

@cro

This comment has been minimized.

Copy link
Member Author

commented Jun 18, 2019

K, fixed the doc test failure I think.

cro added 2 commits Jun 7, 2019
@cro cro force-pushed the cro:stale-events-53411 branch from 4fee5a2 to 9c03423 Jun 18, 2019
@cro

This comment has been minimized.

Copy link
Member Author

commented Jun 18, 2019

And rebased against 2018.3. I think this will be ready for merge if all the tests pass again.

@waynew

This comment has been minimized.

Copy link
Contributor

commented Jun 18, 2019

@cro I think we also need some tests, or is this fixing some existing tests?

@cro

This comment has been minimized.

Copy link
Member Author

commented Jun 18, 2019

I would like to write tests, but according to my understanding of the test suite as it stands:

After banging my head against this for some time I think we will need to resort to a manual test.

  1. The test suite currently starts a master but never restarts one so unless we make this config the default there's no way to alter the configuration
  2. There are no tests (that I can find) in the suite that verify functionality of non-default master config options.

If there is an approach I am unaware of I'm happy to explore it.

Copy link
Member

left a comment

I approve this merge as an exception to the tests writing rule.

cro added a commit that referenced this pull request Jun 25, 2019
Forward port #53412 to 2019.2: Add event_listen_queue_max_seconds to fix #53411
cro added 3 commits Jun 26, 2019
@cro

This comment has been minimized.

Copy link
Member Author

commented Jun 28, 2019

Re-run all

@cro cro merged commit 3270c5b into saltstack:2018.3 Jun 28, 2019
29 of 36 checks passed
29 of 36 checks passed
ci/py2/ubuntu1604 (ignore-me-new-jenkins) This commit cannot be built
Details
ci/py2/windows2016 (ignore-me-new-jenkins) This commit cannot be built
Details
ci/py3/centos7 (ignore-me-new-jenkins) This commit cannot be built
Details
ci/py3/fedora29 (ignore-me-new-jenkins) This commit cannot be built
Details
ci/py3/ubuntu1604 (ignore-me-new-jenkins) This commit cannot be built
Details
ci/py3/ubuntu1804 (ignore-me-new-jenkins) This commit cannot be built
Details
ci/py3/windows2016 (ignore-me-new-jenkins) This commit cannot be built
Details
WIP Ready for review
Details
ci/docs (ignore-me-new-jenkins) This commit looks good
Details
ci/lint (ignore-me-new-jenkins) This commit looks good
Details
ci/py2/centos6 (ignore-me-new-jenkins) This commit looks good
Details
ci/py2/centos7 (ignore-me-new-jenkins) This commit looks good
Details
ci/py2/debian8 (ignore-me-new-jenkins) This commit looks good
Details
ci/py2/debian9 (ignore-me-new-jenkins) This commit looks good
Details
ci/py2/fedora29 (ignore-me-new-jenkins) This commit looks good
Details
ci/py2/ubuntu1804 (ignore-me-new-jenkins) This commit looks good
Details
ci/py3/debian8 (ignore-me-new-jenkins) This commit looks good
Details
ci/py3/debian9 (ignore-me-new-jenkins) This commit looks good
Details
continuous-integration/jenkins/pr-head This commit looks good
Details
jenkins/pr/docs The docs job has passed
Details
jenkins/pr/lint The lint test passed
Details
jenkins/pr/py2-centos-6 The py2-centos-6 job has passed
Details
jenkins/pr/py2-centos-7 The py2-centos-7 job has passed
Details
jenkins/pr/py2-debian-8 The py2-debian-8 job has passed
Details
jenkins/pr/py2-debian-9 The py2-debian-9 job has passed
Details
jenkins/pr/py2-fedora-29 The py2-fedora-29 job has passed
Details
jenkins/pr/py2-ubuntu-1604 The py2-ubuntu-1604 job has passed
Details
jenkins/pr/py2-ubuntu-1804 The py2-ubuntu-1804 job has passed
Details
jenkins/pr/py2-windows-2016 The py2-windows-2016 job has passed
Details
jenkins/pr/py3-centos-7 The py3-centos-7 job has passed
Details
jenkins/pr/py3-debian-8 The py3-debian-8 job has passed
Details
jenkins/pr/py3-debian-9 The py3-debian-9 job has passed
Details
jenkins/pr/py3-fedora-29 The py3-fedora-29 job has passed
Details
jenkins/pr/py3-ubuntu-1604 The py3-ubuntu-1604 job has passed
Details
jenkins/pr/py3-ubuntu-1804 The py3-ubuntu-1804 job has passed
Details
jenkins/pr/py3-windows-2016 The py3-windows-2016 job has passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.