Fortio (greek for load - load test tool) #445

ldemailly · 2017-07-02T06:42:45Z

Command line help:

$ fortio
Φορτίο 0.1 usage:

fortio [flags] url
  -H value
    	Additional Header(s)
  -c int
    	Number of connections/goroutine/threads (0 doesn't change internal default)
  -p string
    	List of pXX to calculate (default "50,75,99,99.9")
  -qps float
    	Queries Per Seconds (default 8)
  -r float
    	Resolution of the histogram lowest buckets in seconds (default 0.001)
  -t duration
    	How long to run the test (default 5s)
  -v int
    	Verbosity level (0 is quiet)

Example output:

$ fortio https://www.google.com
Running at 8 queries per second for 5s: https://www.google.com
Starting at 8 qps with 4 thread(s) [gomax 8] for 5s : 10 calls each (total 40)
2017/07/04 15:49:17 T001 ended after 5.076985609s : 10 calls. qps=1.9696727093874258
2017/07/04 15:49:17 T002 ended after 5.096783364s : 10 calls. qps=1.9620217862569524
2017/07/04 15:49:17 T003 ended after 5.097467191s : 10 calls. qps=1.9617585803506157
2017/07/04 15:49:17 T000 ended after 5.098168545s : 10 calls. qps=1.9614887016254972
Ended after 5.098196115s : 40 calls. qps=7.8459
Sleep times : count 36 avg 0.46568594 +/- 0.0106 min 0.436353622 max 0.482935658 sum 16.7646938
Aggregated Function Time : count 40 avg 0.08815966 +/- 0.01111 min 0.071791544 max 0.118569671 sum 3.5263864
# range, mid point, percentile, count
>= 0.07 < 0.08 , 0.075 , 27.50, 11
>= 0.08 < 0.09 , 0.085 , 60.00, 13
>= 0.09 < 0.1 , 0.095 , 87.50, 11
>= 0.1 < 0.12 , 0.11 , 100.00, 5
# target 50% 0.0869231
# target 75% 0.0954545
# target 99% 0.117084
# target 99.9% 0.118421
Code 200 : 40
Response Body Sizes : count 40 avg 11500.55 +/- 443.3 min 11266 max 12770 sum 460022

This change is

wip

Still wip

Main: - url argument - gather per threads stats and print them Http: - new utility functions for making http calls (only one without 100% test coverage) Periodic: - Factor out thread id calculation in output - don’t use stdout directly, not thread safe - thread index passed to function (so it’s safe to save/use state without locks), always starting at 0 Stats: - Log function that uses the (thread safe) logger for output devel/README.md: - added greek for fortio - my editor removed extra trailing whitespaces

+ make linter happy

fix for connection pooling / running out of socket errors switch from bool debug to int -v verbosity

Semi log fixed buckets

i.e. when we’re falling behind on qps Still todo: merge/aggregation

still todo still is merging (Also some more testing on pXX see TODO)

ldemailly · 2017-07-04T06:56:56Z

left to do:

merging of histogram/counter
-H for additional header (so it works with linkerd) and learn how to have multiples (if possible)
multiple pX (using ...float64)

also (but maybe deferred to future PRs):

a request id in either headers or query arg or both
calculate bytes sent/received including headers (to be researched)

douglas-reid · 2017-07-08T00:08:44Z

devel/fortio/periodic.go, line 166 at r5 (raw file):

Previously, ldemailly (Laurent Demailly) wrote…

missed that comment from jason, will update now (though the caller makes it quite explicit)

ok. i don't think using rate.Limiter is blocking, just wanted to suggest it. I start to get a little nervous when I see custom throttling code, and as you say, getting RPS correct here is a big deal.

Comments from Reviewable

douglas-reid · 2017-07-08T00:13:44Z

devel/fortio/periodic.go, line 174 at r5 (raw file):

Previously, ldemailly (Laurent Demailly) wrote…

I need i changed in the middle and I use it after the loop so that wouldn't work

I assume you mean #453 ? I'm fine if you want to revisit this in a subsequent, more focused PR. I happen to have a select fetish, and think it (or similar) would make the intent here clearer, but this isn't a blocker.

Comments from Reviewable

ldemailly · 2017-07-08T00:15:53Z

devel/fortio/periodic.go, line 174 at r5 (raw file):

#453

yes. though I'd still need to be able to do 40kps on my mac to make the change

Comments from Reviewable

ldemailly · 2017-07-08T00:17:24Z

devel/fortio/periodic.go, line 81 at r7 (raw file):

Previously, ldemailly (Laurent Demailly) wrote…

having an issue making a default RunnerOption constant (or not constant but then the array will be shared or I have to copy() it) - wip

Done.

Comments from Reviewable

ldemailly · 2017-07-08T00:18:11Z

devel/fortio/periodic.go, line 119 at r5 (raw file):

Previously, douglas-reid (Douglas Reid) wrote…

do we have expensive Print clauses in fortio? glog may work for here, even if eval occurs.

#453

Comments from Reviewable

douglas-reid · 2017-07-08T00:24:04Z

devel/fortio/README.md, line 17 at r5 (raw file):

Previously, ldemailly (Laurent Demailly) wrote…

it's a mix of all those - up to GOMAXPROCS it's threads (when running at maxqps at least) and number of simultaneous connections after GOMAXPROCS it's connections but not exactly simultaneous...
the line
	http.DefaultTransport.(*http.Transport).MaxIdleConnsPerHost = numThreads + 1
is what makes it a number of connections (if the call isn't instantaneous)

the tension here is between describing what it does exactly in go terms vs what users expects a load test to do. suggestions welcome.

no great suggestions, unfortunately. just worried about potential user surprise or questions.

Comments from Reviewable

douglas-reid · 2017-07-08T00:25:07Z

devel/fortio/stats.go, line 126 at r5 (raw file):

Previously, ldemailly (Laurent Demailly) wrote…

Histogram is pretty tied to Counter and a close extension (and most people would use just Histogram) but if the file is too big sure (are there rule of thumbs on file size/splitting)

no rule of thumb; up to you.

Comments from Reviewable

ldemailly · 2017-07-08T00:25:29Z

devel/fortio/README.md, line 17 at r5 (raw file):

Previously, douglas-reid (Douglas Reid) wrote…

no great suggestions, unfortunately. just worried about potential user surprise or questions.

np. will update doc as questions come (and also probably the README / godoc). thx

Comments from Reviewable

douglas-reid · 2017-07-08T00:28:56Z

devel/fortio/periodic.go, line 29 at r5 (raw file):

Previously, ldemailly (Laurent Demailly) wrote…

it's just what gets run, not really a callback in traditional sense (even though it is one in the sense of the caller provides it and it gets called (back)) - maybe TestFunction ?

I'm struggling to come up with a better name -- Function just seems too generic. But I won't block, as this isn't important.

Comments from Reviewable

douglas-reid · 2017-07-08T00:31:45Z

devel/fortio/http.go, line 15 at r5 (raw file):

Previously, ldemailly (Laurent Demailly) wrote…

will postpone multi package for a tiny bit if that's ok

as far as I can tell, this layout address my concerns.

Comments from Reviewable

ldemailly · 2017-07-08T00:32:08Z

devel/fortio/periodic.go, line 29 at r5 (raw file):

Previously, douglas-reid (Douglas Reid) wrote…

I'm struggling to come up with a better name -- Function just seems too generic. But I won't block, as this isn't important.

thx ! btw it's not a keyword, and scoped to a package and a type so it's not that generic

Comments from Reviewable

douglas-reid · 2017-07-08T00:32:25Z

Reviewed 1 of 20 files at r1, 16 of 23 files at r6, 1 of 2 files at r9, 2 of 2 files at r10, 4 of 4 files at r11.
Review status: all files reviewed at latest revision, 34 unresolved discussions.

Comments from Reviewable

ldemailly · 2017-07-08T00:32:31Z

devel/fortio/http.go, line 15 at r5 (raw file):

Previously, douglas-reid (Douglas Reid) wrote…

as far as I can tell, this layout address my concerns.

thx!

Comments from Reviewable

douglas-reid · 2017-07-08T00:33:40Z

Cool. I look forward to seeing the regression suite integration.

I think all of my high-level concerns have been addressed or answered. I look forward to the mentioned improvements. Thanks for your patience in getting this review done.

Comments from Reviewable

ldemailly · 2017-07-08T00:35:51Z

thanks a lot for all the thoughtful comments and you patience in educating me about go ! greatly appreciated!

can you accept it in github too ? tia

Review status: all files reviewed at latest revision, 34 unresolved discussions.

Comments from Reviewable

istio-testing · 2017-07-08T00:40:41Z

Jenkins job istio/presubmit passed

douglas-reid

My concerns have been addressed.

ayj · 2017-07-08T00:46:27Z

Reviewed 1 of 20 files at r1, 16 of 23 files at r6, 1 of 2 files at r9, 2 of 2 files at r10, 4 of 4 files at r11.
Review status: 14 of 15 files reviewed at latest revision, 34 unresolved discussions.

Comments from Reviewable

ayj · 2017-07-08T00:50:02Z

Reviewed 1 of 23 files at r6, 1 of 1 files at r12.
Review status: all files reviewed at latest revision, 15 unresolved discussions.

Comments from Reviewable

istio-testing · 2017-07-08T01:09:31Z

Jenkins job istio/presubmit passed

istio-testing · 2017-07-08T01:12:21Z

Jenkins job istio/presubmit passed

* Fortio (greek for load - load test tool) (cmd/fortio/main.go): - url argument - gather per threads stats and print them Http: - new utility functions for making http calls (only one without 100% test coverage) Periodic: - Factor out thread id calculation in output - don’t use stdout directly, not thread safe - thread index passed to function (so it’s safe to save/use state without locks), always starting at 0 Stats: - Log function that uses the (thread safe) logger for output devel/README.md: - added greek for fortio - my editor removed extra trailing whitespaces * Debug dumps full headers * use -c for simulataneous connections / 'threads' fix for connection pooling / running out of socket errors switch from bool debug to int -v verbosity * Adding Histogram Semi log fixed buckets * Calculate % of negative sleep time i.e. when we’re falling behind on qps * Adding test for first and last buckets * Updated defaults + nicer usage string and version * Supports multiple -H for additional Headers * Host header is special * Go ignores Host header with spaces (!) * Validate extra headers early + more efficient + trim * Now aggregating histograms after the threads are done Support for .Clone() .Transfer() .Reset() .CopyFrom() in Counter/Histogram Aggregate data Reduced default verbosity (use -v 1 to see per thread stats and body sizes histogram) * Added -p for list of percentiles, -r for resolution... Added -p for list of percentiles, -r for resolution. Also added very first rough cut at some doc/README * Even better when actually using the resolution arg Added a more meaty example, will add picture later * adding a picture * New maxqps mode (-qps 0) and low overhead echo server IPeriodicRunner w Setter/Getter -> just NewPeriodicRunner(params *RunnerOptions) thanks Mandar and Doug for the hint ! Also pass the r pointer for most things in runOne() Also added tests for maxqps and parse percentiles. Coverage 81% (missing are mostly fatals and verbose > 1) * Code review changes Thanks a lot Doug and Jason for all the comments * Moved files to symmetrical layout library at top level and command lines under cmd/ If we split into multiple packages we can move it under fortio/pkg/fortio but that seems ugly right now [2x fortio in the path, though it’s there for cmd/fortio/…]) * Improve runOne param name/self doc * Factor the defaults out per code review suggestions <rant>wish there was const for struct (and pointers)</rant> * Fixed readme links Thx for the review * code reviews comments + use glog in echosrv * Further (last ;-) ?) set of code review suggestions (godoc, Options(), if...) * Dropping last defer Former-commit-id: 2bcb4dc

istio#445) * networking: Restrict port match in match condition to use port numbers only Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com> * Clarify semantics of hostnames, ports, etc Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com> * nits Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com> * nits Signed-off-by: Shriram Rajagopalan <shriramr@vmware.com>

* Remove disabled components when running mesh manifest apply. * Add workarounds for pruning component when getting empty manifests. * Fix GetAll() func comment. * Fix status update logic in reconciler when manifests are empty. * Update with some comments and catch an edge case. * Add tests for kubectl Delete() and GetAll(). * Refactor client.go to extract the common code of kubectl execution. * Add comments for kubectlParams, and improve comments for kubectl(). * Polish the log and comments. * Remove redundant comments on kubectlParams.

Fortio (greek for load - load test tool)

dc5f4ed

wip

googlebot added the cla: yes label Jul 2, 2017

ldemailly self-assigned this Jul 2, 2017

ldemailly added 8 commits July 2, 2017 01:32

Now using multiple threads

df9fa5e

Still wip

Debug dumps full headers

78599bc

+ make linter happy

use -c for simulataneous connections / 'threads'

5bc75c3

fix for connection pooling / running out of socket errors switch from bool debug to int -v verbosity

Adding Histogram

ac10271

Semi log fixed buckets

Calculate % of negative sleep time

9a6af10

i.e. when we’re falling behind on qps Still todo: merge/aggregation

Merge branch 'master' into ldemailly-bench4

082cdbd

Adding test for first and last buckets

c3bac65

still todo still is merging (Also some more testing on pXX see TODO)

ldemailly added 5 commits July 4, 2017 00:21

Updated defaults + nicer usage string and version

2c97567

Supports multiple -H for additional Headers

847c0cb

Host header is special

062f621

Go ignores Host header with spaces (!)

de23981

Validate extra headers early + more efficient + trim

4a80dec

istio deleted a comment from istio-testing Jul 4, 2017

douglas-reid approved these changes Jul 8, 2017

View reviewed changes

Dropping last defer

097895e

ayj approved these changes Jul 8, 2017

View reviewed changes

ldemailly merged commit 2bcb4dc into master Jul 8, 2017

ldemailly deleted the ldemailly-bench4 branch July 8, 2017 01:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fortio (greek for load - load test tool) #445

Fortio (greek for load - load test tool) #445

ldemailly commented Jul 2, 2017 •

edited

ldemailly commented Jul 4, 2017 •

edited

douglas-reid commented Jul 8, 2017

douglas-reid commented Jul 8, 2017

ldemailly commented Jul 8, 2017 •

edited

ldemailly commented Jul 8, 2017

ldemailly commented Jul 8, 2017

douglas-reid commented Jul 8, 2017

douglas-reid commented Jul 8, 2017

ldemailly commented Jul 8, 2017

douglas-reid commented Jul 8, 2017

douglas-reid commented Jul 8, 2017

ldemailly commented Jul 8, 2017

douglas-reid commented Jul 8, 2017

ldemailly commented Jul 8, 2017

douglas-reid commented Jul 8, 2017

ldemailly commented Jul 8, 2017

istio-testing commented Jul 8, 2017

douglas-reid left a comment

ayj commented Jul 8, 2017

ayj commented Jul 8, 2017

istio-testing commented Jul 8, 2017

istio-testing commented Jul 8, 2017

Fortio (greek for load - load test tool) #445

Fortio (greek for load - load test tool) #445

Conversation

ldemailly commented Jul 2, 2017 • edited

ldemailly commented Jul 4, 2017 • edited

douglas-reid commented Jul 8, 2017

douglas-reid commented Jul 8, 2017

ldemailly commented Jul 8, 2017 • edited

ldemailly commented Jul 8, 2017

ldemailly commented Jul 8, 2017

douglas-reid commented Jul 8, 2017

douglas-reid commented Jul 8, 2017

ldemailly commented Jul 8, 2017

douglas-reid commented Jul 8, 2017

douglas-reid commented Jul 8, 2017

ldemailly commented Jul 8, 2017

douglas-reid commented Jul 8, 2017

ldemailly commented Jul 8, 2017

douglas-reid commented Jul 8, 2017

ldemailly commented Jul 8, 2017

istio-testing commented Jul 8, 2017

douglas-reid left a comment

Choose a reason for hiding this comment

ayj commented Jul 8, 2017

ayj commented Jul 8, 2017

istio-testing commented Jul 8, 2017

istio-testing commented Jul 8, 2017

ldemailly commented Jul 2, 2017 •

edited

ldemailly commented Jul 4, 2017 •

edited

ldemailly commented Jul 8, 2017 •

edited