Stream client's PutMessages() doesn't fully utilize retry policies.

[rcaputo]

streaming's PutMessages() is not retrying when it fails due to OCI reporting a 429 error.

This is because OCI reports 429s at the PutMessagesResponse.Entries[].Error level. The top level PutMessagesResponse.RawResponse.StatusCode is 200.

For example, when I put 750 messages in a single PutMessagesRequest bundle, I receive seven hundred and fifty 429 errors wrapped in a "200 OK" envelope.

I'd like to present the possibility that the active retry policy should be invoked in this situation.

One way might be for the SDK to hoist the per-message error to the RawResponse level if every message fails the same way.

Another would be for OCI to respond with an HTTP 429 instead of 200.

% time go run . --persec 750 --perput 750
2024-02-09T12:05:35.919114162-05:00 : putting 750
200 OK
2024-02-09T12:05:36.373495651-05:00 : putting 750
200 OK
2024-02-09T12:05:36.709358229-05:00 : putting 750
200 OK
0 : 429 : Request was throttled because bytes limit exhausted, next request can be made in 453 ms
1 : 429 : Request was throttled because bytes limit exhausted, next request can be made in 453 ms
2 : 429 : Request was throttled because bytes limit exhausted, next request can be made in 453 ms
3 : 429 : Request was throttled because bytes limit exhausted, next request can be made in 453 ms
4 : 429 : Request was throttled because bytes limit exhausted, next request can be made in 453 ms
...
745 : 429 : Request was throttled because bytes limit exhausted, next request can be made in 453 ms
746 : 429 : Request was throttled because bytes limit exhausted, next request can be made in 453 ms
747 : 429 : Request was throttled because bytes limit exhausted, next request can be made in 453 ms
748 : 429 : Request was throttled because bytes limit exhausted, next request can be made in 453 ms
749 : 429 : Request was throttled because bytes limit exhausted, next request can be made in 453 ms
exit status 1
go run . --persec 750 --perput 750  0.61s user 0.17s system 46% cpu 1.669 total

[rcaputo]

REST documentation for PutMessages describes a perfectly reasonable error 429.

Oddly, the RawMessage headers include a Retry-After header. This is documented as part of status 429 but not 200. At this point, RawMessage contains elements from two different response types. It remains to be seen whether the REST API or the SDK is introducing the discrepancy.

The SDK is correctly invoking retry logic, but there's no reason to retry while the RawMessage holds a status code 200.