apiserver: rate-limit logsink receives

[axw]

Description of change

Update the logsink handler so that it will
rate-limit receipt of log messages. Rate-limiting
defaults to a burst of 1000 messages, after
which we allow 1 message per millisecond. This
is configurable by modifying the controller
agent's agent.conf. If we find that users
need to configure it regularly, we can promote
the configuration to controller config.

QA steps

1. juju bootstrap localhost
2. juju model-config -m controller logging-config='=TRACE'
3. juju ssh -m controller 0 "sudo tail -F /var/log/juju/logsink.log"

Observe that the logs come in quickly.

4. juju ssh -m controller 0
5. edit /var/lib/juju/agents/machine-0/agent.conf, adding LOGSINK_RATELIMIT_REFILL: 1s under values.
6. sudo service jujud-machine-0 restart

Observe that the log messages come in quickly to begin with (up to 1000 messages), and then they are limited to 1 per second.

Documentation changes

Probably not until we know that people need to tweak the defaults, at which point we can add controller-config.

Bug reference

None.

[mjs]

This looks great. Thanks.

How did you decide on the default rate limiting values?

[mjs]

It might be nicer if LogSink was a pointer. Then if it's nil then the defaults could be used. This might be preferable to having to specify the defaults all over the place.

[axw]

I've done that in my other PR that makes the db logging parameters configurable. I'll rebase this once that has landed.

[axw]

How did you decide on the default rate limiting values?

Pretty arbitrarily; 1000 log messages/s per agent seems like quite a lot already to me. I tested this with canonical-kubernetes, and it didn't cause any backing up. I'll put it in the cards to do some scale testing before we release 2.2.1.

[axw]

$$merge$$

[jujubot]

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju

[jujubot]

Build failed: Generating tarball failed
build url: http://juju-ci.vapour.ws:8080/job/github-merge-juju/11153

[axw]

$$merge$$

[jujubot]

Status: merge request accepted. Url: http://juju-ci.vapour.ws:8080/job/github-merge-juju