Refactor h2load rate mode options

[tatsuhiro-t]

h2load rate mode works great.
The only downside is it require some thinking about how to specify the combination of -n, -C, -r, -m.
Also -m is originally the number of concurrent streams per HTTP/2 or SPDY session, that is the protocol detail configuration. So we'd better not to use it in the way we now use it in rate mode.

I'm thinking a bit simpler configuration options, which can be used very straight forward.
But be aware that this is not backward compatible to the current rate mode.

Here is my current plan.

Without rate mode, we have no changes.
With rate mode, we only use the following options to determine the number of requests, clients, rate etc:

• -x: new option. This is the number of requests made per connection.
• -r: This is the number of connections made per --rate-period (h2load: Add option for user-definable rate period #380)
• -C: This is the maximum number of connections made.

We already have -r and -C. -x is new addition. In rate mode, -n and -c options are ignored.
To remove the test time conflicts, -C option is now mandatory if -r is used. -x defaults to "auto", which has the same meaning of the default value of -m option.
Instead of adding -x, we can say that -n option means the number of requests made per connection in rate mode.

What do you think about this?

[tatsuhiro-t]

Perhaps, instead of adding -x, we use -n as it is now and divide by -C, which is now required option for rate mode, and calculation is trivial.

[tatsuhiro-t]

More simplification can be made to remove -C option, and use -c option instead.

[tatsuhiro-t]

So if we call the first plan A, then plan B is:

With rate mode, we only use the following options to determine the number of requests, clients, rate etc:

• -n: This is the total number of requests made (not per connection)
• -r: This is the number of connections made per --rate-period (h2load: Add option for user-definable rate period #380)
• -c: This is the maximum number of connections made.

Here -c has a bit of different meaning without rate mode, but in both mode, it specifies maximum number of clients, so very similar. If we specify -r option, h2load enters rate mode.
These 3 options are required. They have default values, so it is not necessarily required to be in command-line. But most likely you have to specify -c if we use -r > 1.
The number of requests made per connection is simply -n / -c.

[tatsuhiro-t]

#383 implements plan B.
@nshoemaker @LPardue Do you agree with this change? Do we lose much of the split of original rate mode if we do this?

[LPardue]

PR looks good. I don't think this affects how I would use h2load (other than tweaking the commands). However, I'd like to check my understanding with this change by looking at an example.

Consider a script file containing 7 URI lines;

a) h2load --timing-script-file=7.uris -c2 would send a total of 14 requests?
b) h2load --timing-script-file=7.uris -c2 -n7 would send a total of 7 requests? (How are the requests split between clients?)
c) h2load --timing-script-file=7.uris -c2 -n14 would send a total of 14 requests?
d) h2load --timing-script-file=7.uris -c2 -n21 would send a total of 14 requests? (Value 21 is ignored and clamped to 14?)
e) h2load --timing-script-file=7.uris -c2 -r1 would send a total of 14 requests?
f) h2load --timing-script-file=7.uris -t2 -c2 -r2 would send a total of 14 requests?
g) h2load --timing-script-file=7.uris -t2 -c6 -r1 syntax error the connection rate must be greater than or equal to the number of threads
h) h2load --timing-script-file=7.uris -t2 -c6 -r2 would send a total of 42 requests?
i) h2load --timing-script-file=7.uris -t2 -c6 -r2 -n7 would send a total of 7 requests? (How are the requests split between clients?)
j) h2load --timing-script-file=7.uris -t2 -c6 -r2 -n5 syntax error the number of requests must be greater than or equal to the concurrent clients.

[tatsuhiro-t]

a) h2load --timing-script-file=7.uris -c2 would send a total of 14 requests?

Yes

b) h2load --timing-script-file=7.uris -c2 -n7 would send a total of 7 requests? (How are the requests split between clients?)

Yes.
The requests are split into 2 clients, so one client gets 4 and another gets 3.
First client uses first 4 URIs in script file.
2nd client uses first 3 URIs in script file.

c) h2load --timing-script-file=7.uris -c2 -n14 would send a total of 14 requests?

Yes

d) h2load --timing-script-file=7.uris -c2 -n21 would send a total of 14 requests? (Value 21 is ignored and clamped to 14?)

Yes

e) h2load --timing-script-file=7.uris -c2 -r1 would send a total of 14 requests?

Yes

f) h2load --timing-script-file=7.uris -t2 -c2 -r2 would send a total of 14 requests?

Yes.

g) h2load --timing-script-file=7.uris -t2 -c6 -r1 syntax error the connection rate must be greater than or equal to the number of threads

yes

h) h2load --timing-script-file=7.uris -t2 -c6 -r2 would send a total of 42 requests?

yes

i) h2load --timing-script-file=7.uris -t2 -c6 -r2 -n7 would send a total of 7 requests? (How are the requests split between clients?)

yes. 7 requests were split up into 6 clients as: 1, 2, 1, 1, 1, 1.

j) h2load --timing-script-file=7.uris -t2 -c6 -r2 -n5 syntax error the number of requests must be greater than or equal to the concurrent clients.

yes.

[LPardue]

Then it meets all my expectations 👍

[tatsuhiro-t]

Thank you!

I have another idea about timing script and -n option handling.
Since timing script is special in a sense that we'd like to execute all URIs per client, it may be easier to treat -n option specially to specify the number of request per client if --timing-script option is given.
Then users don't have to do multiplication by hand.

So if we do this:

b) h2load --timing-script-file=7.uris -c2 -n7 would send a total of 7 requests? (How are the requests split between clients?)

will send a total of 14 requests, since -n7 applies to each client.

[LPardue]

Since timing script is special in a sense that we'd like to execute all URIs per client, it may be easier to treat -n option specially to specify the number of request per client if --timing-script option is given.
Then users don't have to do multiplication by hand.

This sounds good, it is more fair to always have the clients issue the same number of URIs. The hand calculations can be tricky and easy to get wrong, especially when iterating on test configurations (I know because my old implementation of script required me to do this).

-n is really useful for scripting as it allows a user to splice the script file between 1 and n, helping when doing development testing. etc.

[tatsuhiro-t]

OK, let's do this.

[tatsuhiro-t]

As discussed above, a5070ba implements special handling of -n option if it is used with --timing-script-file option

[tatsuhiro-t]

If everything is ok, I will squash into one commit and merge PR into master.

[LPardue]

I pulled down your branch to do some local testing.

It looks like -r is split across -t. It think it would help to make this clear in doc for -r option? First example looks ok.

hload -t2 --timing-script-file='7.uris' -n7 -r2 -c8

starting benchmark...
spawning thread #0: 4 total client(s). Up to 1 client(s) will be created every 1 seconds. 28 total requests
spawning thread #1: 4 total client(s). Up to 1 client(s) will be created every 1 seconds. 28 total requests
Application protocol: h2c

finished in 14.06s, x.x req/s, y.yMB/s
requests: 56 total, 56 started, 56 done, 56 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 56 2xx, 0 3xx, 0 4xx, 0 5xx
...

This example looks weird?

hload -t2 --timing-script-file='7.uris' -n7 -r4 -c8
starting benchmark...
spawning thread #0: 4 total client(s). 28 total requests
spawning thread #1: 4 total client(s). 28 total requests
Application protocol: h2c

finished in 16.58s, x.x req/s, y.yMB/s
requests: 56 total, 56 started, 56 done, 56 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 56 2xx, 0 3xx, 0 4xx, 0 5xx

[tatsuhiro-t]

Thank you. If you meant the absence of "Up to 2 client(s) will ..." in the second example, it was fixed in 13d03f5.
-r distribution among threads are documented in b2b1748.

[LPardue]

Thanks! Looks good!

[tatsuhiro-t]

Merged as 0d27a89

[nshoemaker]

This looks great!