fix: reduce the probability of RESOURCE_EXHAUSTED errors during tests #734
Conversation
Reduces the probability of RESOURCE_EXHAUSTED errors during tests by making the GetOperation method retry errors with this code with an exponential backoff. The GetOperation method is called repeatedly for long-running operations by a polling future. These calls also count towards the max 5 admin requests per second. Fixes #733
product-auto-label
bot
added
the
api: spanner
google-cla
bot
added
the
cla: yes
| callSettings = | ||
| callSettings | ||
| .toBuilder() | ||
| .setRetryableCodes(codes) | ||
| .build(); |
There was a problem hiding this comment.
| callSettings = | |
| callSettings | |
| .toBuilder() | |
| .setRetryableCodes(codes) | |
| .build(); | |
| callSettings = callSettings.toBuilder().setRetryableCodes(codes).build(); |
| ImmutableSet.<StatusCode.Code>builderWithExpectedSize( | ||
| callSettings.getRetryableCodes().size() + 1) | ||
| .addAll(callSettings.getRetryableCodes()) | ||
| .add(StatusCode.Code.RESOURCE_EXHAUSTED) |
There was a problem hiding this comment.
It would be preferable if this could be changed in the generated code, but I don't know whether that is possible considering this comes from the generic com.google.longrunning.stub.OperationsStub.
There was a problem hiding this comment.
It seems we would need this only for fixing the tests (correct me if I am wrong). If that is the case, changing production behaviour to fix those might not be the best path forward. Could we inject different behaviour in test only.
On another note, do we still need this change now that the tests seem to be working as expected?
There was a problem hiding this comment.
I've updated the PR to only change this behavior if SpannerOptions#setAutoThrottleAdministrativeRequests has been set. This means that it will only be enabled for (integration) tests.
Codecov Report
@@ Coverage Diff @@
## master #734 +/- ##
============================================
- Coverage 85.15% 85.03% -0.12%
- Complexity 2562 2563 +1
============================================
Files 142 143 +1
Lines 13960 13983 +23
Branches 1331 1335 +4
============================================
+ Hits 11887 11890 +3
- Misses 1513 1532 +19
- Partials 560 561 +1
Continue to review full report at Codecov.
|
…#734) * fix: reduce the probability of RESOURCE_EXHAUSTED errors during tests Reduces the probability of RESOURCE_EXHAUSTED errors during tests by making the GetOperation method retry errors with this code with an exponential backoff. The GetOperation method is called repeatedly for long-running operations by a polling future. These calls also count towards the max 5 admin requests per second. Fixes #733 * fix: use default retry settings * chore: run formatter * fix: only retry RESOURCE_EXHAUSTED when throttling requests
Reduces the probability of
RESOURCE_EXHAUSTEDerrors during tests by making the GetOperation method retry errors with this code with an exponential backoff. The GetOperation method is called repeatedly for long-running operations by a polling future. These calls also count towards the max 5 admin requests per second.Fixes #733