Implement Resource Consumer

[piosz]

This is @socaa's intern project.

Background

We are working on autoscaling for Kubernetes in 3 dimensions:

• cluster size autoscaling
• horizontal autoscaling of pod - changing the size of replication controller WIP: auto-scaler proposal #2863
• vertical autoscaling of pod - changing its resource limits Vertical pod auto-sizer #10782

We need a tool which will help us to test e2e those kinds of autoscaling.

Resource Consumer

Resource Consumer will consist of two parts: container consuming resources and library managing this load.

Container

The container which consumes specified amount of resources (cpu/mem). Docker container, uploaded to Google Container Registry, open sourced, possibly written in go (it’s not a requirement, but it’s strongly encouraged). The resources to consume should be set and mutated by sending appropriate http request to the container.

The interface should allow to consume the given amount of cpu/mem for the given period of time. It should consist of the following methods:

ConsumeCPU(milicores, duration_sec)
ConsumeMem(megabytes, duration_sec)
GetCurrentStatus()

Each http request will be handled by spawning a new process which will consume given amount of resource and exit after given timeout.The request cannot be cancelled.

Consuming CPU

Consuming the whole core can be done easily. The problem is how to consume some part of it. One of possible ways to achieve it is to split time into small quants, then perform heavy compute operation during some percentage of quants and sleep during the other ones. It is a part of the task to figure out the best way to consume CPU.

Consuming memory

There are few possible ways how to handle memory consumption. We want to make sure the memory can be correctly freed. The only problem is that the container use some memory itself, so that this amount should be considered in calculations as well. It is a part of the task to figure out the best way to consume memory.

Library

Client side library written in go will be linked with e2e tests and should consist of two kinds of methods: static consumption of resources and dynamic one.

Static consumption

This part of library should allow to consume constant amount of resource over time:

ConsumeCPU(node, percentage)
ConsumeMem(node, percentage)
CreateConsumingPod(node) : PodID
ConsumeCPU(pod, percentageOfRequest)
ConsumeMem(pod, percentageOfRequest)

Dynamic consumption

This part of library should allow to create service on top of replication controller, and then request to consume the given amount of resources by all replicas together based on service level load balancing. It can be implemented with sending a lot of consumption request for short period of time and small amount of resources. The interface should consist of:

CreateConsumingService() : RcID
ConsumeCPU(rc, milicores)
ConsumeMem(rc, megabytes)

In both cases there might be an option to add possibility to modify/stop consumption if needed.

Use cases

Cluster size autoscaling

1. consume more resources on each node that is specified for autoscaler
2. observe that cluster size increased

Horizontal autoscaling of pod

1. create consuming RC and start consuming appropriate amount of resources
2. observe that RC has been resized
3. observe that usage on each replica decreased

Vertical autoscaling of pod

1. create consuming pod and start consuming appropriate amount of resources
2. observed that limits has been increased

Milestones

• Resource Consumer application
  • http API skeleton server Added http API skeleton server. #12076
    • define API
    • implement mock methods which just return theirs names
  • implement cpu consumption Added consume cpu function to Resource Consumer #12517
  • implement memory consumption Memory consumption added to Resource Consumer #13789
  • figure out how to verify that the resource consumer works correctly and do it
  • create Dockerfile, build and deploy Docker image Add docker file to Resource Consumer #12257
  • [optional] accept resource.Quantity style strings in request
• Client library for e2e test
  • static consumption Static Consumption added to autoscaling_utils.go #14167
  • dynamic consumption using Kubernetes service based load balancing Dynamic cpu consumption #13243, Memory dynamic consumption #14036
• Implement few e2e test for autoscaling using Resource Consumer to prove it is useful
  • port horizontal cluster autoscaling tests Use Resource Consumer for tests in autoscaling.go #14275
  • horizontal pod autoscaling based on cpu Horizontal Pod Autoscaling e2e tests #13640
  • horizontal pod autoscaling based on memory Memory dynamic consumption #14036
• Ensure that tests are not flaky.
• Document how to use Resource Consumer added README file to Resource Consumer #14645
• Final presentation

Dropped ideas

• [optional] add possibility to specify initial consumption request
• [optional] add a status page

Releases

Date	Version	Image	Release notes
8/14/15	alpha	gcr.io/google_containers/resource_consumer:alpha	Support only for cpu consumption
9/15/15	beta	gcr.io/google_containers/resource_consumer:beta	Feature complete
12/14/15	beta2	gcr.io/google_containers/resource_consumer:beta2	Added support for custom metrics

[piosz]

cc @jszczepkowski @fgrzadkowski

[wojtek-t]

/cc me

[bgrant0607]

cc @AnanyaKumar

[vishh]

/sub

[derekwaynecarr]

/sub

[ramr]

/sub

[piosz]

The alpha version is released as gcr.io/google_containers/resource_consumer:alpha. It supports consuming cpu only.

[piosz]

The beta version is released as gcr.io/google_containers/resource_consumer:beta. It supports consuming both cpu and memory.

cc @derekwaynecarr

[piosz]

Good job @socaa!