Retry on interrupted connections when reading from S3 in native FS library

[nineinchnick]

Description

Read the response inside a transformer lambda and throw a retryable
exception on IO errors to use the already configured SDK retry mechanism.

Additional context and related issues

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

[nineinchnick]

The solution is based on aws/aws-sdk-java#856 (comment)

[SemionPar]

LGTM!

[electrum]

Testing this kind of stuff is hard. I like the usage of Toxiproxy. It's very cool that we can wrap this around an arbitrary service. The test verifies that retries are happening, which is good enough, so the below is just a thought/suggestion.

It would be nice to test that reads succeed after the retry. Given that S3 is simple and the request/response pattern is also simple, using MockWebServer might be a good alternative. You could enqueue multiple failure responses for the failure case, and failure+success responses for the success-with-retry case.

[mayankvadariya]

🎉

[nineinchnick]

@electrum I first tried doing this with the MockSyncHttpClient from the AWS SDK testing services library, but without constructing a complete/proper request, the SDK was doing retries automatically before calling the response transformer. I'd much rather implement a custom toxic that'd only drop the first N connections. WDYT?

Maybe that can be a follow-up, I applied all comments and this should be ready to go.