Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transaction retries could timeout and propagate the Aborted error to the client application #1008

Closed
olavloite opened this issue Mar 23, 2021 · 0 comments · Fixed by #1009
Closed
Assignees
Labels
api: spanner priority: p2 type: bug

Comments

@olavloite
Copy link
Contributor

@olavloite olavloite commented Mar 23, 2021

Read/write transactions are automatically retried by the client library if the transaction is aborted by the backend. These retries use the backoff settings that are returned by the backend, or if no backoff settings are returned, the default backoff settings of the Rollback RPC. These default settings can however also cause transactions that are retried multiple times to fail with an Aborted error, if the total time that the retries are executed exceed the default total timeout of the Rollback RPC (30 seconds).

@olavloite olavloite added priority: p2 type: bug labels Mar 23, 2021
@olavloite olavloite self-assigned this Mar 23, 2021
@product-auto-label product-auto-label bot added the api: spanner label Mar 23, 2021
olavloite added a commit that referenced this issue Mar 23, 2021
Transactions that are retried because of an aborted transaction use the
retry settings of the Rollback RPC. This ensures reasonable backoff values.
It however also meant that transactions that are retried multiple times
could exceed the total timeout of the retry settings, and that again would
cause the Aborted error to propagate. This change sets the total timeout
for transaction retries to 24 hours and disables any max attempts in the
retry settings to prevent retries to fail because the deadline is exceeded.

Transactions can still fail with timeout errors if individual RPC invocations
exceed the configured timeout of that RPC. This change only prevents timeouts
from occurring because of repeated retries of an entire transaction.

Fixes #1008
olavloite added a commit that referenced this issue Mar 24, 2021
Transactions that are retried because of an aborted transaction use the
retry settings of the Rollback RPC. This ensures reasonable backoff values.
It however also meant that transactions that are retried multiple times
could exceed the total timeout of the retry settings, and that again would
cause the Aborted error to propagate. This change sets the total timeout
for transaction retries to 24 hours and disables any max attempts in the
retry settings to prevent retries to fail because the deadline is exceeded.

Transactions can still fail with timeout errors if individual RPC invocations
exceed the configured timeout of that RPC. This change only prevents timeouts
from occurring because of repeated retries of an entire transaction.

Fixes #1008
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: spanner priority: p2 type: bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant