Add a global locate sample rate, with optional rate controller

[jwhitlock]

For issue #1398, add a global sample rate for locate observations. Automatically adjust it, with the goal of processing as many observations as possible without accumulating a growing backlog (and without manual intervention).

To make the data queues easier to monitor, the queue metrics has two new tags:

• queue_type: data or task
• data_type: For data queues, the type of data. bluetooth, cell, and (most of all) wifi contribute to the backlog calculation

DataQueue and derived classes store the data_type, passed as a new initialization variable.

The queue metric task monitor_queue_size is now monitor_queue_size_and_rate_control, and runs the rate controller if enabled. The rate controller is implemented by a PID controller provided by the simple-pid library. Simulation showed that only proportional control was needed, so it is more of a P Controller.

This PR adds some new Redis keys:

• global_locate_sample_rate: Read in web app, assumed 100.0 (100%) if unset. Set in the task app, if the rate controller is enabled
• rate_controller_target: The target maximum data queue size, as an integer. Must be set by an administrator to enable the rate controller.
• rate_controller_enabled: Read in the task app, 1 to enable the rate controller, 0 or unset to disable. Set by an administrator.
• rate_controller_kp, rate_controller_ki, rate_controller_kd: K_p, K_i, and K_d, the proportional, integral, and derivative gain terms. They are set to defaults (8, 0, and 0) when the rate controller is enabled, and could be adjusted by an administrator.
• rate_controller_state: The internal state of the PID controller, as a JSON-encoded string, used to reload it when the task runs.

There are new metrics as well:

• rate_control.locate: The current value of global_locate_sample_rate, or 100.0 if unset
• rate_control.locate.kp, rate_control.locate.ki, rate_control.locate.kd: The current values of PID gains K_p, K_i, and K_d.
• rate_control.locate.pterm, rate_control.locate.iterm, rate_control.locate.dterm: The internal components of the PID controller, for debugging and adjusting the PID gains.

There are new documents for rate control, as well as updates to the metrics docs.

[jwhitlock]

I'm a little bothered by the way the rate jumps around when under rate control. This could be solved by a filter on the rate output (some suggest a second PID controller on the output). However, it is more important to get some real-world data from using this code, so stopping here.

Locally, I ran:

• make build docs lint test, standard QA stuff
• Three terminals:
  • make run for the web server. No change is expected.
  • make runcelery for the backend. The new tags for the queues are emitted.
  • make shell so I could run redis-cli -h redis

The rate controller can't be tested outside of simulations (in private docs and private repositories, since it uses production data for simulated traffic), but I was able to manually test:

• Set rate_controller_enabled to 1, is reset to 0 when the queue task runs.
• Set global_locate_sample_rate to 50, rate_controller_target to 1000, rate_controller_enabled to 1. When the queue task runs, the rate controller runs, it sets the rate to 100.0, sets the gains to the defaults, and emits the metrics.

Let's get some real-world usage data.