Add resources requests and limits for all containers

[jpmcb]

Issue number:

Closes #240

Description of changes:

This patch provides the expected resource (cpu and memory) requests for each container along with some reasonable limits. These numbers were derived from several experiments (noted in #240 and in the process of creating this patch): the 10m CPU and 50Mi memory limit is due to components having spikes in resource requirements when nodes rotate and pods are re-scheduled (during upgrade scenarios where the agent, api-server, and/or controller exist on the upgrading node). The request delimiter allows for more resources to be consumed (if they are available on the node) and should be safe since we drain the nodes before doing an upgrade. Having too tight of resource limits will cause the system's kernel to kill the process, so these are intentionally forgiving.

TODO:

• documentation for consumers of brupop

Testing done:

Several integration tests were run with the metric-server and kubernetes dashboard also installed to monitor resource utilization.

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[jpmcb]

Force pushed to add documentation ✏️

This PR is ready to go, but leaving in draft for now so we don't get in the way of the upcoming v1.0.0 release

[jpmcb]

Marking ready for review now that 1.0.0 is out and force pushed to address a merge conflict in the README.md documentation!

[jpmcb]

Final verification experiment:

1. Installed the kubernetes metrics server
2. Installed the kubernetes dashboard with an admin user
3. Ran integration tests off this patch using the modified yaml manifest and the default 1.0.0image.

Results

API server

Memory below 3Mi
CPU at 1.00 M

Controller

Memory stable ~2.75 Mi with small spikes below 3 Mi
CPU at 1.00 M or lower

Agents

Memory below 7 Mi
CPU at 1.00 M with small spikes up to ~3 M during node updates