Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce benchmarking for PRs #739

Open
ivanvc opened this issue Apr 24, 2024 · 6 comments
Open

Introduce benchmarking for PRs #739

ivanvc opened this issue Apr 24, 2024 · 6 comments

Comments

@ivanvc
Copy link
Member

ivanvc commented Apr 24, 2024

Formalizing the comment from PR #691 (comment). Adding a GitHub action or Prow Job to compare benchmarks vs. main would help catch issues sooner, like the one tracked in #720.

@ahrtr, a couple of questions to finalize scoping this task:

  • What would be a good set of benchmark parameters to run?
  • Should this run on GitHub Actions or as a Prow Job?
  • What would be the priority for this task?
@ahrtr
Copy link
Member

ahrtr commented May 13, 2024

Thanks @ivanvc for raising this issue. Please see my response below,

  • What would be a good set of benchmark parameters to run?

For the writing test, I think we should commit the TXN at least multiple times, i.e 10 times. So options.Iterations/options.BatchSize > 10. For the options.BatchSize, it should be big enough, i.e. at least 10K.

  • Should this run on GitHub Actions or as a Prow Job?

We can add it on github action firstlyh. Eventually we may want to migrate all workflow checks to Prow.

What would be the priority for this task?

It may not be a urgent task.

@fuweid
Copy link
Member

fuweid commented May 13, 2024

Eventually we may want to migrate all workflow checks to Prow.

Is it any reason to move workflow to Prow?

@ahrtr
Copy link
Member

ahrtr commented May 13, 2024

It's part of the effort of kubernetes/k8s.io#6102.

At least the Prow has more powerful machine, we can have shorter running time for the performance test. Please see #691

@fuweid
Copy link
Member

fuweid commented May 13, 2024

It's part of the effort of kubernetes/k8s.io#6102.

At least the Prow has more powerful machine, we can have shorter running time for the performance test. Please see #691

I was thinking that github action can provide VM which supports nested virtualization and we can simulate real power-failure. And it's more easy to debug CI changes in GitHub action. Just my two cents. Anyway, thanks for sharing the background.

@ivanvc
Copy link
Member Author

ivanvc commented Jun 28, 2024

The initial implementation in #750 is benchmarking only the default write and read modes (write=rnd, read=seq).

@ahrtr, should we expand the benchmarks to other combinations?

@ahrtr
Copy link
Member

ahrtr commented Jun 29, 2024

@ahrtr, should we expand the benchmarks to other combinations?

Yes, it's a good point. etcd is actually sequentially writing the keys. We need to consider different write modes and read modes, also different key & value sizes. Usually K8s's value size is bigger than 1K+? Could you draft a simple doc /doc/benchmark.md firstly? thx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

3 participants