Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Progress bar for benchmark tests #196

Closed
wants to merge 9 commits into from
Closed

Conversation

jeffbean
Copy link
Collaborator

Fixes #57

Prints a Progress bar to the console based on request count. Using library that does atomic increments. Other notes is it uses the terminal width and we are not setting it manually.

Output:

yab moe --health --rps 100 -d 10s

{
  "body": {
    "result": {
      "ok": true
    }
  },
  "ok": true,
  "trace": "4802f563e101ee42"
}

Benchmark parameters:
  CPUs:            24
  Connections:     48
  Concurrency:     1
  Max requests:    1000
  Max duration:    10s
  Max RPS:         100
 1000 / 1000 [============================================================================================================================] 100.00% 10s
Benchmark finished
Latencies:
  0.5000: 785.466µs
  0.9000: 1.006528ms
  0.9500: 1.189203ms
  0.9900: 9.582816ms
  0.9990: 57.667579ms
  0.9995: 59.614299ms
  1.0000: 61.56102ms
Elapsed time:      10s
Total requests:    1000
RPS:               100.00

Test without finishing total requests yab moe --health --rps 20000 -d 10s

{
  "body": {
    "result": {
      "ok": true
    }
  },
  "ok": true,
  "trace": "78915b6b9d43168c"
}

Benchmark parameters:
  CPUs:            24
  Connections:     48
  Concurrency:     1
  Max requests:    200000
  Max duration:    10s
  Max RPS:         20000
 98210 / 200000 [===========================================================>-------------------------------------------------------------]  49.10% 10s
Benchmark finished
Latencies:
  0.5000: 565.675µs
  0.9000: 1.575297ms
  0.9500: 13.889164ms
  0.9900: 87.541209ms
  0.9990: 91.826491ms
  0.9995: 93.020593ms
  1.0000: 101.400753ms
Elapsed time:      10.045s
Total requests:    98210
RPS:               9776.14

@mention-bot
Copy link

@jeffbean, thanks for your PR! By analyzing the history of the files in this pull request, we identified @prashantv, @billf and @kriskowal to be potential reviewers.

benchmark.go Outdated
@@ -100,6 +104,7 @@ func runBenchmark(out output, allOpts Options, m benchmarkMethod) {
if opts.RPS > 0 && opts.MaxDuration > 0 {
// The RPS * duration in seconds may cap opts.MaxRequests.
rpsMax := int(float64(opts.RPS) * opts.MaxDuration.Seconds())
// FIXME: if we are using small time like 500ms the calculation is still 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm why is that? Seconds is a float, so it should be a positive integer

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if RPS and MaxDuration are both low enough, the product of multiplying them together will be a fraction less than 1. e.g. 10 * time.Millisecond.Seconds() = 0.01 and the cast will truncate int(0.01) = 0

https://play.golang.org/p/VrpN9zrZW4

benchmark.go Outdated
@@ -132,6 +137,9 @@ func runBenchmark(out output, allOpts Options, m benchmarkMethod) {
states[i] = newBenchmarkState(statter)
}

progressBar := pb.StartNew(opts.MaxRequests)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's 2 main types of benchmarks:

  • by total number of requests
  • by total duration

we're only taking total number of requests into account but we should also think about duration.

we could progress bars for both if both are specified?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. Found a race testing this with Duration. After playing around with the library I forked it and might try to upstream more fixes. Seems small enough and active that this can wait for the upstream to take my fix or we can use my fork and improve the library for our needs as well.

cheggaaa/pb#103

@@ -16,6 +16,7 @@ import:
version: ^1
- package: github.com/uber/jaeger-client-go
version: ^1
- package: gopkg.in/cheggaaa/pb.v1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the github repo supports semver, maybe better to go directly with that and ^1

benchmark.go Outdated
@@ -100,6 +104,7 @@ func runBenchmark(out output, allOpts Options, m benchmarkMethod) {
if opts.RPS > 0 && opts.MaxDuration > 0 {
// The RPS * duration in seconds may cap opts.MaxRequests.
rpsMax := int(float64(opts.RPS) * opts.MaxDuration.Seconds())
// FIXME: if we are using small time like 500ms the calculation is still 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if RPS and MaxDuration are both low enough, the product of multiplying them together will be a fraction less than 1. e.g. 10 * time.Millisecond.Seconds() = 0.01 and the cast will truncate int(0.01) = 0

https://play.golang.org/p/VrpN9zrZW4

@bufdev
Copy link
Contributor

bufdev commented Oct 26, 2017

@jeffbean what is the status on this?

@jeffbean
Copy link
Collaborator Author

This lib keeps introducing data races. i will come back to this when I fix more upstream issues.

@jeffbean jeffbean closed this Oct 26, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants