Increasing padding percentage before repacking will cause a failure

[billonahill]

If a topology is deployed with a given padding percentage for the packing plan and later a scaling event occurs with a lower padding percentage set, the repack will fail to initialize the new PackingPlan in the PackingPlanBuilder.initContainers() method.

We need to investigate to see if this is really an issue. If this setting is persisted in the protobuf topology configs on the state server, and immutable for the lifecycle of the topology, we're ok and there is no issue. If that's not the case, we could have a bug if the value is unset and DEFAULT_CONTAINER_PADDING_PERCENTAGE in ResourceCompliantRRPacking or FirstFitDecreasingPacking changes, or if the value is set in a config and modified after topology submission. This is the logic used at initialization time by the packing implementations.

    int paddingPercentage = TopologyUtils.getConfigWithDefault(topologyConfig,
        TOPOLOGY_CONTAINER_PADDING_PERCENTAGE, DEFAULT_CONTAINER_PADDING_PERCENTAGE);

Here's why this setting (and possibly other defaults) causes a problem:

When intitializing PackingPlan Containers without an existing packing plan (e.g., via the pack() method), PackingPlan.initContainers() creates containers using the configured this.maxContainerResource value.

When intitializing PackingPlan Containers with an existing packing plan (e.g., via the repack() method), PackingPlan.initContainers() creates containers using the currentPackingPlan.getMaxContainerResources() value which uses the largest container size found in the existing packing (i.e., the one from the initial packing during submission). This creates problems when adding back the instnances to re-initialize the container, since the padding percentage is used when asserting container capacity.

For example, say we put 4 instances each using 1GB ram into a container with 0% padding. The container size would be 4GB.

Later when we go to repack, we reconstruct the packing plan and we have 10% padding set. The container size would be taken as the largest found, so 4GB. When we put the 4th instance into that container it would fail, since container capacity if 4GB, but we assert that there is room for 4GB of resource with an addition 10% headroom, which doesn't exist.

At a minimum we should better clarify what the runtime expectation should be when settings like padding change during before repacking.