New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node Performance Testing. #65249

Open
balajismaniam opened this Issue Jun 19, 2018 · 6 comments

Comments

Projects
None yet
4 participants
@balajismaniam
Member

balajismaniam commented Jun 19, 2018

Background

  1. We have some low level node performance optimization features already implemented in the Kubelet (e.g., CPU manager).
  2. We don't have a way to objectively measure how well certain performance-sensitive applications run on k8s across versions.
  3. We don’t have performance data on how performance-sensitive applications benefit from the already implemented features.
  4. We don't have a way to measure the effect of further performance features on the classes of performance sensitive applications that our users care about. This leads to making decisions based on speculative analysis of performance impact, or inability to make decisions.
  5. In March 2018 at the latest resource management working group face-to-face meeting, we agreed to prioritize building a node performance testing framework and suite over some of the performance optimization proposals. This way we can make better decisions based on objective data.

Goals

  1. Detect performance regressions.
  2. Provide objective data to guide node performance optimization features.

Proposal Document

https://docs.google.com/document/d/14SfsL1Bs5jEniNaYN7kgRan9m5ZhuyLPBPvTrZ1mK94/edit#

Phase 1 Action/Exploration Items

  1. Node e2e test driver.
  2. Determine initial set of workloads.
  3. Add workload container images for node performance testing.
  4. Documentation.
  5. Periodically run the node perf testing suite with initial set of workloads.
  6. Represent node performance test results in one of the test dashboards.
    1. Choose either https://k8s-testgrid.appspot.com/ or http://node-perf-dash.k8s.io.
    2. Initial result will be FAILED or PASSED.
      1. Use some simple stats (e.g., the test failed if performance of workload X falls outside of -5% performance from the average performance of the last 10 runs of the workload).
    3. Resources
      1. Node-perf-dash: https://github.com/kubernetes/contrib/tree/master/node-perf-dash
      2. Testgrid: https://github.com/kubernetes/test-infra/tree/master/testgrid
      3. Compare between two test runs: https://github.com/kubernetes/perf-tests/tree/master/compare.
      4. How PASSED or FAILED is posted now: https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/gubernator.sh#L152-L157
      5. Do we need to use NodePerfData type: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/perftype/perftype.go#L38-L45

Pull Requests

  1. Adding docker images for workloads: #65251.
  2. The node performance testing framework: #65250.

/assign @balajismaniam @ConnorDoyle.
CC @vishh @derekwaynecarr

@balajismaniam

This comment has been minimized.

Show comment
Hide comment
@balajismaniam
Member

balajismaniam commented Jun 19, 2018

/sig node
/assign @balajismaniam @ConnorDoyle

@tpepper

This comment has been minimized.

Show comment
Hide comment
@tpepper

tpepper Jun 20, 2018

Contributor

Will this target the 1.12 milestone?

Contributor

tpepper commented Jun 20, 2018

Will this target the 1.12 milestone?

k8s-merge-robot added a commit that referenced this issue Sep 1, 2018

Merge pull request #65251 from balajismaniam/node-perf-images
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.

Add docker images for node perf testing workloads.

This PR adds docker images for the workloads used in node performance testing.
Partially fixes: #65249

CC @ConnorDoyle 

This PR has been tested in the following environment:
- n1-standard-16
- Ubuntu 16.04
- docker 17.03.2
@balajismaniam

This comment has been minimized.

Show comment
Hide comment
@balajismaniam

balajismaniam Sep 5, 2018

Member

#65251 is one of the steps.
/reopen

Member

balajismaniam commented Sep 5, 2018

#65251 is one of the steps.
/reopen

@k8s-ci-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-ci-robot

k8s-ci-robot Sep 5, 2018

Contributor

@balajismaniam: Reopening this issue.

In response to this:

#65251 is one of the steps.
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Contributor

k8s-ci-robot commented Sep 5, 2018

@balajismaniam: Reopening this issue.

In response to this:

#65251 is one of the steps.
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot reopened this Sep 5, 2018

k8s-merge-robot added a commit that referenced this issue Sep 8, 2018

Merge pull request #65250 from balajismaniam/node-perf-testing-framework
Automatic merge from submit-queue (batch tested with PRs 65250, 68241). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md.

Initial node performance testing framework.

This PR adds a framework for node performance testing. 
Partially fixes: #65249.
Use the following command to run this test:
```sh
make test-e2e-node FOCUS="Node Performance Testing" SKIP="" PARALLELISM=1
```
It has been tested in the following environment:
- n1-standard-16
- Ubuntu 16.04
- docker 17.03.2

Note to reviewers:
This PR won't pass node e2e since the docker images in #65251 are required for this to function. The node e2e will fail when trying to pull the required images for testing.
@balajismaniam

This comment has been minimized.

Show comment
Hide comment
@balajismaniam

balajismaniam Sep 10, 2018

Member

/reopen

Member

balajismaniam commented Sep 10, 2018

/reopen

@k8s-ci-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-ci-robot

k8s-ci-robot Sep 10, 2018

Contributor

@balajismaniam: Reopening this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Contributor

k8s-ci-robot commented Sep 10, 2018

@balajismaniam: Reopening this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot reopened this Sep 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment