Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: reduce the probability of RESOURCE_EXHAUSTED errors during tests #734

Merged
merged 6 commits into from Dec 16, 2020

Conversation

olavloite
Copy link
Contributor

@olavloite olavloite commented Dec 13, 2020

Reduces the probability of RESOURCE_EXHAUSTED errors during tests by making the GetOperation method retry errors with this code with an exponential backoff. The GetOperation method is called repeatedly for long-running operations by a polling future. These calls also count towards the max 5 admin requests per second.

Fixes #733

olavloite added 2 commits Dec 13, 2020
Reduces the probability of RESOURCE_EXHAUSTED errors during tests by making the
GetOperation method retry errors with this code with an exponential backoff.
The GetOperation method is called repeatedly for long-running operations by a
polling future. These calls also count towards the max 5 admin requests per second.

Fixes #733
@olavloite olavloite requested a review from thiagotnunes Dec 13, 2020
@olavloite olavloite requested a review from as a code owner Dec 13, 2020
@product-auto-label product-auto-label bot added the api: spanner label Dec 13, 2020
@google-cla google-cla bot added the cla: yes label Dec 13, 2020
callSettings =
callSettings
.toBuilder()
.setRetryableCodes(codes)
.build();

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
callSettings =
callSettings
.toBuilder()
.setRetryableCodes(codes)
.build();
callSettings = callSettings.toBuilder().setRetryableCodes(codes).build();

ImmutableSet.<StatusCode.Code>builderWithExpectedSize(
callSettings.getRetryableCodes().size() + 1)
.addAll(callSettings.getRetryableCodes())
.add(StatusCode.Code.RESOURCE_EXHAUSTED)
Copy link
Contributor Author

@olavloite olavloite Dec 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be preferable if this could be changed in the generated code, but I don't know whether that is possible considering this comes from the generic com.google.longrunning.stub.OperationsStub.

Copy link
Contributor

@thiagotnunes thiagotnunes Dec 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems we would need this only for fixing the tests (correct me if I am wrong). If that is the case, changing production behaviour to fix those might not be the best path forward. Could we inject different behaviour in test only.

On another note, do we still need this change now that the tests seem to be working as expected?

Copy link
Contributor Author

@olavloite olavloite Dec 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated the PR to only change this behavior if SpannerOptions#setAutoThrottleAdministrativeRequests has been set. This means that it will only be enabled for (integration) tests.

@codecov
Copy link

@codecov codecov bot commented Dec 14, 2020

Codecov Report

Merging #734 (76bcbec) into master (aa701f5) will decrease coverage by 0.11%.
The diff coverage is 18.51%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #734      +/-   ##
============================================
- Coverage     85.15%   85.03%   -0.12%     
- Complexity     2562     2563       +1     
============================================
  Files           142      143       +1     
  Lines         13960    13983      +23     
  Branches       1331     1335       +4     
============================================
+ Hits          11887    11890       +3     
- Misses         1513     1532      +19     
- Partials        560      561       +1     
Impacted Files Coverage Δ Complexity Δ
...rpcDatabaseAdminStubWithCustomCallableFactory.java 0.00% <0.00%> (ø) 0.00 <0.00> (?)
...m/google/cloud/spanner/spi/v1/GapicSpannerRpc.java 81.35% <20.00%> (-2.04%) 81.00 <0.00> (ø)
...ud/spanner/SessionPoolAsyncTransactionManager.java 87.30% <0.00%> (+1.58%) 13.00% <0.00%> (+2.00%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aa701f5...76bcbec. Read the comment docs.

@thiagotnunes thiagotnunes merged commit cd946d7 into master Dec 16, 2020
19 of 21 checks passed
@thiagotnunes thiagotnunes deleted the reduce-resource-exhausted-errors branch Dec 16, 2020
@thiagotnunes thiagotnunes mentioned this pull request Dec 17, 2020
thiagotnunes pushed a commit that referenced this issue May 6, 2021
…#734)

* fix: reduce the probability of RESOURCE_EXHAUSTED errors during tests

Reduces the probability of RESOURCE_EXHAUSTED errors during tests by making the
GetOperation method retry errors with this code with an exponential backoff.
The GetOperation method is called repeatedly for long-running operations by a
polling future. These calls also count towards the max 5 admin requests per second.

Fixes #733

* fix: use default retry settings

* chore: run formatter

* fix: only retry RESOURCE_EXHAUSTED when throttling requests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: spanner cla: yes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants