Fix lease manager next check scheduling #13429
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
scheduled lease manager block check. This causes a missed check if scheduled at a zero duration from now.
have the potential for concurrent access to it.
SimonRichardson
approved these changes
Oct 18, 2021
|
|
Merged
jujubot
added a commit
that referenced
this pull request
Oct 19, 2021
#13430 Merge 2.9 #13419 Recognise VXLAN network devices #13423 Fix add-machine command help for zones constraint #13422 Provision OpenStack instances with NICs for bindings #13427 Update to latest Pebble #13429 Fix lease manager next check scheduling #13428 Various txn fixes mainly for mongo 4.4 compatibility Conflicts due to mongo changes already in develop ``` # Conflicts: # cloudconfig/podcfg/image.go # cloudconfig/podcfg/podcfg.go # featuretests/upgrade_test.go # mongo/mongo.go # testing/mgo.go # ``` ## QA steps See PRs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A missing assignment in the lease manager means that we can decide that the next block check is now, rather than at least a second from now.
This was not causing us issues until we added a tiny efficiency gain regarding scheduling the check. Before we would schedule the next check if it was now, otherwise we would not.
In addition to the linked bug, an intermittent test failure is possible, observed here.
Here we fix the assignment so that a second is correctly added to the next check time.
Even with this fix applied, we would often not schedule a next check due to the fact that we try to avoid walking back the next check later than one we have scheduled. This logic is flawed if we just made a check and are calculating the next. This too is fixed.
QA steps
For the intermittent test failure, run
TestLeadershipNoLeaseBlockEvaluatedNextTickin loop with-race. The failure is very hard to trigger, but the test is fast an will run a good number of times in a reasonable duration.For practical QA:
juju model-config -m controller logging-config="<root>=INFO;juju.worker.lease.raft=TRACE".juju debug log -m controller.Documentation changes
None.
Bug reference
https://bugs.launchpad.net/juju/+bug/1942354
https://bugs.launchpad.net/juju/+bug/1947409
https://bugs.launchpad.net/juju/+bug/1947179