Propagate serialization IOException instead of rethrowing as runtime

[jack-berg]

Related to #6069. Alternative to #6076.

cc @ricardo-mestre

I verified the claim in #6076 by simulating IOExceptions while serializing. The threads in the OkHttp dispatcher thread pool were continually dying and being replaced due to IOExceptions being rethrown as RuntimeExceptions. This PR fixes that by adjusting the contract of HttpSender to accept a Marshaler instead of a Consumer<OutputStream>. While Consumer<OutputStream> was forced to catch exceptions, the Marshaler throws IOExceptions.

Its hard to recreate this behavior in a test, but I did verify in a contrived local setup that this change no longer results in the OkHttp dispatcher threads dying.

Note to @ricardo-mestre - this PR doesn't change the definition for which exceptions are retryable. That's defined here and needs followup in a separate PR. In contrast to OkHttpHttpSender, the JdkHttpSender currently retrying on almost all IOExceptions as defined here. Were able to do this because a user provided some great real-world data of all the different IOExceptions encountered and virtually all of them were retryable.

[codecov]

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (517ef88) 90.95% compared to head (9a70faf) 90.99%.
Report is 2 commits behind head on main.

Files	Patch %	Lines
...ry/exporter/sender/jdk/internal/JdkHttpSender.java	75.00%	1 Missing and 1 partial ⚠️
...telemetry/exporter/internal/http/HttpExporter.java	50.00%	1 Missing ⚠️
...orter/sender/okhttp/internal/OkHttpHttpSender.java	85.71%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #6082      +/-   ##
============================================
+ Coverage     90.95%   90.99%   +0.04%     
- Complexity     5682     5687       +5     
============================================
  Files           628      628              
  Lines         16672    16675       +3     
  Branches       1651     1654       +3     
============================================
+ Hits          15164    15174      +10     
+ Misses         1051     1047       -4     
+ Partials        457      454       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ricardo-mestre]

Thanks a lot your active support with this one !

[jkwatson]

could we use awaitility for this, rather than result.join?

Something like await().untilAsserted(() -> Assertions.assertThat(result.isSuccess()).isTrue());

[jack-berg]

I think in cases like this where the future is expected to resolve quickly and deterministically, its ok to use join. For example, we use it all over the place in AbstractHttpTelemetryExporterTest.

With that said, I'm actually not sure what this new test has to do with this change so I've gone ahead and removed it.