Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add rate limiter and update tests. #10987

Merged
merged 7 commits into from Dec 5, 2019

Conversation

howbazaar
Copy link
Contributor

@howbazaar howbazaar commented Dec 4, 2019

Add configurable rate limiting for agent connections.

This work took a sideways step as I needed to write tests for the rate limiting, but all the login tests were using JujuConnSuite. So I changed the login tests to instead be based on statetetesting.StateSuite, which is much lighter.

QA steps

juju bootstrap lxd test
juju controller-config agent-ratelimit-max=5
juju controller-config agent-ratelimit-rate=500ms
juju deploy ~jameinel/ubuntu-lite -n 5
for i in {1..5}; do juju add-unit ubuntu-lite -n 5 --to 0,1,2,3,4; done
watch -c -n 1 juju status --color
# in a different shell 
juju ssh -m controller 0
sudo service jujud-machine-0 restart
# observe the reduced rate of reconnection in the status 

Documentation changes

Will need to document the two new controller config values.

@howbazaar howbazaar marked this pull request as ready for review December 5, 2019 02:35
@howbazaar howbazaar force-pushed the 2.7-apiserver-agent-ratelimiting branch from a5ad4d9 to 62b195c Compare December 5, 2019 02:49
@howbazaar howbazaar force-pushed the 2.7-apiserver-agent-ratelimiting branch from 62b195c to 6b63be1 Compare December 5, 2019 02:51
@howbazaar
Copy link
Contributor Author

I need to update this to support disabling the ratelimiting.

@hpidcock
Copy link
Member

hpidcock commented Dec 5, 2019

LGTM. Works well, tested the rate limiting on microk8s with ~50 agents.

Copy link
Member

@jameinel jameinel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this overall, I would like to see an experiment with 1000+ agents so we are comfortable with the behavior with and without this change.

@@ -29,29 +29,31 @@ func (s *ControllerSuite) TestControllerAndModelConfigInitialisation(c *gc.C) {
c.Assert(err, jc.ErrorIsNil)

optional := set.NewStrings(
controller.IdentityURL,
controller.IdentityPublicKey,
controller.AgentRateLimitMax,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this is just Sort() the list alphabetically

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I hit my OCD limit

srv.updateAgentRateLimiter(data.Config)
})
if err != nil {
logger.Criticalf("programming error in subscribe function: %v", err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Criticalf in this code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is critical by design.

@@ -534,6 +534,8 @@ func (s *JujuConnSuite) setUpConn(c *gc.C) {
for key, value := range s.ControllerConfigAttrs {
s.ControllerConfig[key] = value
}
// Explicitly disable rate limiting.
s.ControllerConfig[controller.AgentRateLimitMax] = 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

@howbazaar
Copy link
Contributor Author

$$merge$$

@howbazaar
Copy link
Contributor Author

$$merge$$

1 similar comment
@howbazaar
Copy link
Contributor Author

$$merge$$

@jujubot jujubot merged commit 8090882 into juju:2.7 Dec 5, 2019
@howbazaar howbazaar deleted the 2.7-apiserver-agent-ratelimiting branch December 5, 2019 22:22
@manadart manadart mentioned this pull request Dec 9, 2019
jujubot added a commit that referenced this pull request Dec 10, 2019
#11001

Merge 2.7 forward for:
- #10996 from wallyworld/lookup-action-prefix
- #10995 from anastasiamac/missing-cons-lp1854510
- #10968 from anastasiamac/add-cred-UR-force
- #10987 from howbazaar/2.7-apiserver-agent-ratelimiting
- #10992 from howbazaar/2.7-test-races
- #10933 from jameinel/2.7-delay-done
- #10966 from jameinel/2.7-handle-start-error-1854338
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants