Fix HTTP/2: retry requests rejected with REFUSED_STREAM

[magurotuna]

This patch gets the client to retry HTTP/2 requests rejected with REFUSED_STREAM by inspecting the error detail.

I found the issue with highly concurrent requests in HTTP/2 made to the server that is configured to allow the smaller number of concurrent streams. So for example, a client makes, say, 500 concurrent requests to an HTTP/2 server while the server's max_concurrent_streams is set to 100. In this case the client is allowed to initiate 500 concurrent streams until the initial SETTINGS frame from the server reaches to the client, though the server will reject 400 streams by responding REFUSED_STREAM.
From the standpoint of users of reqwest crate, I would expect the crate to deal with this kind of situation behind the scenes. Actually, as far as I searched, Go and libcurl provides the retry logic when they encounter REFUSED_STREAM. Considering the "ergonomic, batteries-included" characteristic of reqwest, I believe automatic retry would be very nice to have.

Here's a repo containing reproducible code written in a couple different languages. It also has rust-patched code which demonstrates that this patch does fix the issue.
https://github.com/magurotuna/deno-fetch-h2-stream-error-repro

[seanmonstar]

Thanks for this! I think this might be something we can do, I'll have to go carefully read the spec about REFUSED_STREAM again. But something I was wondering: assuming that this is a safe thing to retry, could we have a unit test for it? Probably just creating a server with a low max concurrent stream limit, and send a bunch of simple requests. It might mean the support test server might need to pass some options to hyper...

[magurotuna]

Sure, I'll add a unit test!

[magurotuna]

Added two test cases in f6ad3c6.
I added two, because I found that if we use the existing test server, it just passed this test without the retry logic - after some investigation, it turned out that the hyper server is so fast that we can't reproduce a scenario that I saw when using nginx as a server, that is, server's initial SETTINGS frame reaches the client later than the client having already opened more streams than is allowed.
In order to get a test case that fails without the retry, the new server has been added that can simulate a "slow" server. I confirmed that the test case using this "slow" server does fail without the retry logic.

[magurotuna]

A related topic; thinking a bit further about the fact that we get REFUSED_STREAM errors when making too many concurrent requests, doing retry is certainly a mitigation, but another approach would be that we start with the relatively small number of concurrent streams until the server's initial SETTINGS frame arrives. This should reduce the chance of getting rejected with REFUSED_STREAM in the first place.
I've opened a proposal issue in h2 (hyperium/h2#731). But I believe retry is still good to have also.

[magurotuna]

Saw CI failed due to missing h2::Error::is_reset method. To address it I bumped h2 crate to 0.3.14 as it was supported since this version.

[magurotuna]

Sorry for making many commits to pass the CI. I made a small fix in 895f282 (the port 19999 was introduced mistakenly in f6ad3c6 for me to easily debug it).
I locally confirmed that cargo nextest --locked --workspace have passed.

[magurotuna]

Thank you for starting the CI many times! I now understand how the test suite works against many feature flags.