From c2ecc4f12279c91768d73b090983e9079af9a64a Mon Sep 17 00:00:00 2001 From: Antti Kervinen Date: Tue, 9 Aug 2022 15:36:24 +0300 Subject: [PATCH] docs: improve balloons documentation - Describe how the policy works. - Document relevant configuration parameters. - Add CPU classes. --- docs/policy/balloons.md | 150 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 144 insertions(+), 6 deletions(-) diff --git a/docs/policy/balloons.md b/docs/policy/balloons.md index 2ead369a8..8b1478353 100644 --- a/docs/policy/balloons.md +++ b/docs/policy/balloons.md @@ -3,11 +3,47 @@ ## Overview The balloons policy implements workload placement into "balloons" that -are pools of exclusive CPUs. A balloon can be inflated and deflated, -that is CPUs added and removed, based on the CPU resource requests of -the workloads in the balloon. The policy supports both static and -dynamically created and popped balloons. The balloons policy enables -configuring balloon-specific CPU classes. +are disjoint CPU pools. Balloons can be inflated and deflated, that is +CPUs added and removed, based on the CPU resource requests of +containers. Balloons can be static or dynamically created and +destroyed. CPUs in balloons can be configured, for example, by setting +min and max frequencies on CPU cores and uncore. + +## How It Works + +1. User configures balloon types from which the policy instantiates + balloons. + +2. A balloon has a set of CPUs and a set of containers that run on the + CPUs. + +3. Every container is assigned to exactly one balloon. A container is + allowed to use all CPUs of its balloon and no other CPUs. + +4. Every logical CPU belongs to at most one balloon. There can be CPUs + that do not belong to any balloon. + +5. The number of CPUs in a balloon can change during the lifetime of + the balloon. If a balloon inflates, that is CPUs are added to it, + all containers in the balloon are allowed to use more CPUs. If a + balloon deflates, the opposite is true. + +6. When a new container is created on a Kubernetes node, the policy + first decides the type of the balloon that will run the + container. The decision is based on annotations of the pod, or the + namespace if annotations are not given. + +7. Next the policy decides which balloon of the decided type will run + the container. Options are: + - an existing balloon that already has enough CPUs to run its + current and new containers + - an existing balloon that can be inflated to fit its current and + new containers + - new balloon. + +9. When a CPU is added to a balloon or removed from it, the CPU is + reconfigured based on balloon's CPU class attributes, or idle CPU + class attributes. ## Deployment @@ -23,6 +59,69 @@ system of CRI-RM. See [setup and usage](../setup.md#setting-up-cri-resource-manager) for more details on managing the configuration. +### Parameters + +Balloons policy parameters: + +- `PinCPU` controls pinning a container to CPUs of its balloon. The + default is `true`: the container cannot use other CPUs. +- `PinMemory` controls pinning a container to the memories that are + closest to the CPUs of its balloon. Pinning memory disallows using + memory from other NUMA nodes. +- `IdleCPUClass` specifies the CPU class of those CPUs that do not + belong to any balloon. +- `ReservedPoolNamespaces` is a list of namespaces (wildcards allowed) + that are assigned to the special reserved balloon, that is, will run + on reserved CPUs. This always includes the `kube-system` namespace. +- `BalloonTypes` is a list of balloon type definitions. Each type can + be configured with the following parameters: + - `Name` of the balloon type. This is used in pod annotations to + assign containers to balloons of this type. + - `Namespaces` is a list of namespaces (wildcards allowed) whose + pods should be assigned to this balloon type, unless overridden by + pod annotations. + - `MinBalloons` is the minimum number of balloons of this type that + is always present, even if the balloons would not have any + containers. The default is 0: if a balloon has no containers, it + can be destroyed. + - `MaxCPUs` specifies the maximum number of CPUs in any balloon of + this type. Balloons will not be inflated larger than this. 0 means + unlimited. + - `MinCPUs` specifies the minimum number of CPUs in any balloon of + this type. When a balloon is created or deflated, it will always + have at least this many CPUs, even if containers in the balloon + request less. + - `CpuClass` specifies the name of the CPU class according to which + CPUs of balloons are configured. + - `PreferSpreadingPods`: if `true`, containers of the same pod + should be spread to different balloons of this type. The default + is `false`: prefer placing containers of the same pod to the same + balloon(s). + - `PreferPerNamespaceBalloon`: if `true`, containers in the same + namespace will be placed in the same balloon(s). On the other + hand, containers in different namespaces are preferrably placed in + different balloons. The default is `false`: namespace has no + effect on choosing the balloon of this type. + - `PreferNewBalloons`: if `true`, prefer creating new balloons over + placing containers to existing balloons. This results in + preferring exclusive CPUs, as long as there are enough free + CPUs. The default is `false`: prefer filling and inflating + existing balloons over creating new ones. + - `AllocatorPriority` (0: High, 1: Normal, 2: Low, 3: None). CPU + allocator parameter, used when creating new or resizing existing + balloons. If there are balloon types with pre-created balloons + (`MinBalloons` > 0), balloons of the type with the highest + `AllocatorPriority` are created first. + +Related configuration parameters: +- `policy.ReservedResources.CPU` specifies the (number of) CPUs in the + special `reserved` balloon. By default all containers in the + `kube-system` namespace are assigned to the reserved balloon. +- `cpu.classes` defines CPU classes and their parameters (such as + `minFreq`, `maxFreq`, `uncoreMinFreq` and `uncoreMaxFreq`). + +### Example + Example configuration that runs all pods in balloons of 1-4 CPUs. ```yaml policy: @@ -30,18 +129,57 @@ policy: ReservedResources: CPU: 1 balloons: + PinCPU: true + PinMemory: true + IdleCPUClass: lowpower BalloonTypes: - Name: "quad" MinCpus: 1 MaxCPUs: 4 + CPUClass: dynamic Namespaces: - "*" +cpu: + classes: + lowpower: + minFreq: 800 + maxFreq: 800 + dynamic: + minFreq: 800 + maxFreq: 3600 + turbo: + minFreq: 3000 + maxFreq: 3600 + uncoreMinFreq: 2000 + uncoreMaxFreq: 2400 ``` See the [sample configmap](/sample-configs/balloons-policy.cfg) for a complete example. -### Debugging +## Assigning a Container to a Balloon + +The balloon type of a container can be defined in pod annotations. In +the example below, the first annotation sets the balloon type (`BT`) +of a single container (`CONTAINER_NAME`). The last two annotations set +the default balloon type for all containers in the pod. + +```yaml +balloon.balloons.cri-resource-manager.intel.com/container.CONTAINER_NAME: BT +balloon.balloons.cri-resource-manager.intel.com/pod: BT +balloon.balloons.cri-resource-manager.intel.com: BT +``` + +If a pod has no annotations, its namespace is matched to the +`Namespaces` of balloon types. The first matching balloon type is +used. + +If the namespace does not match, the container is assigned to the +special `default` balloon, that means reserved CPUs unless `MinCPUs` +or `MaxCPUs` of the `default` balloon type are explicitely defined in +the `BalloonTypes` configuration. + +## Metrics and Debugging In order to enable more verbose logging and metrics exporting from the balloons policy, enable instrumentation and policy debugging from the