spanner.cloud-client.backup_sample_test: test_update_backup failed #3241

flaky-bot · 2020-03-31T20:27:08Z

This test failed!

To configure my behavior, see the Build Cop Bot documentation.

If I'm commenting on this issue too often, add the buildcop: quiet label and
I will stop commenting.

commit: 7bc4783
buildURL: Build Status, Sponge
status: failed

The text was updated successfully, but these errors were encountered:

flaky-bot · 2020-04-01T01:14:41Z

Test passed for commit 9e64bff (Build Status, Sponge)! Closing this issue.

flaky-bot · 2020-04-01T18:54:44Z

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

commit: e943bff
buildURL: Build Status, Sponge
status: failed

flaky-bot · 2020-04-22T11:59:05Z

Oops! Looks like this issue is still flaky. 😬

I reopened the issue, but a human will need to close it again.

commit: c71f978
buildURL: Build Status, Sponge
status: failed

flaky-bot · 2020-04-24T11:51:23Z

Oops! Looks like this issue is still flaky. 😬

I reopened the issue, but a human will need to close it again.

commit: 86a552c
buildURL: Build Status, Sponge
status: failed

tmatsuo · 2020-04-25T03:20:03Z

This is a rare backend issue. We may need to have a mitigation if this happens a lot, but for now we close this.

flaky-bot · 2020-04-27T12:15:12Z

Oops! Looks like this issue is still flaky. 😬

I reopened the issue, but a human will need to close it again.

commit: 2b74710
buildURL: Build Status, Sponge
status: failed

tmatsuo · 2020-04-27T17:42:27Z

It seems like the default timeout for create_backup is 30 seconds.

We can add timeout parameter to create_backup call for mitigating the test failure, but it is also too long for the server to return a Long Running Operation. If the server is expected to take more than 30 seconds frequently enough (currently it seems like so), we should change the client library's default value.

@larkee WDYT?

Update: There is a new config for the timeout. I filed googleapis/python-spanner#68

tmatsuo · 2020-04-27T21:01:25Z

Blocked by googleapis/python-spanner#68

flaky-bot · 2020-05-06T13:33:14Z

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

commit: 32777eb
buildURL: Build Status, Sponge
status: failed

Test output

Traceback (most recent call last):
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "{"created":"@1588771706.167677943","description":"Error received from peer ipv4:108.177.112.95:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):

File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/backup_sample_test.py", line 97, in test_update_backup

backup_sample.update_backup(INSTANCE_ID, BACKUP_ID)

File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/backup_sample.py", line 232, in update_backup

backup.update_expire_time(new_expire_time)

File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/cloud/spanner_v1/backup.py", line 249, in update_expire_time

api.update_backup(backup_update, update_mask, metadata=metadata)

File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/cloud/spanner_admin_database_v1/gapic/database_admin_client.py", line 1192, in update_backup

return self._inner_api_calls["update_backup"](

File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 143, in call

return wrapped_func(*args, **kwargs)

File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/retry.py", line 281, in retry_wrapped_func

return retry_target(

File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/retry.py", line 184, in retry_target

return target()

File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout

return func(*args, **kwargs)

File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable

six.raise_from(exceptions.from_grpc_error(exc), exc)

File "", line 3, in raise_from

google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded

tmatsuo · 2020-05-06T16:26:39Z

The last build failure is:

Traceback (most recent call last):
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "{"created":"@1588771706.167677943","description":"Error received from peer ipv4:108.177.112.95:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/backup_sample_test.py", line 97, in test_update_backup
    backup_sample.update_backup(INSTANCE_ID, BACKUP_ID)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/backup_sample.py", line 232, in update_backup
    backup.update_expire_time(new_expire_time)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/cloud/spanner_v1/backup.py", line 249, in update_expire_time
    api.update_backup(backup_update, update_mask, metadata=metadata)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/cloud/spanner_admin_database_v1/gapic/database_admin_client.py", line 1192, in update_backup
    return self._inner_api_calls["update_backup"](
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
    return wrapped_func(*args, **kwargs)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/retry.py", line 281, in retry_wrapped_func
    return retry_target(
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/retry.py", line 184, in retry_target
    return target()
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
    return func(*args, **kwargs)
  File "/tmpfs/src/github/python-docs-samples/spanner/cloud-client/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded

tmatsuo · 2020-05-06T17:26:02Z

The current deadline is 30 seconds for UpdateBackup.

I'll talk with the Spanner folks to decide how to handle this case.

tmatsuo · 2020-05-07T17:33:58Z

The spanner team is going to bump the deadline, so we don't have any action items other than wait for the next release.

tmatsuo · 2020-05-27T13:55:48Z

Got the new release, closing

flaky-bot bot added buildcop: issue priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Mar 31, 2020

blunderbuss-gcf bot assigned leahecole Mar 31, 2020

flaky-bot bot closed this as completed Apr 1, 2020

flaky-bot bot reopened this Apr 1, 2020

flaky-bot bot added the buildcop: flaky label Apr 1, 2020

leahecole assigned mmmarklu and unassigned leahecole Apr 1, 2020

JustinBeckwith added the api: spanner Issues related to the Spanner API. label Apr 2, 2020

yoshi-automation added the 🚨 This issue needs some love. label Apr 17, 2020

tmatsuo closed this as completed Apr 21, 2020

flaky-bot bot reopened this Apr 22, 2020

tmatsuo mentioned this issue Apr 23, 2020

[spanner] fix: bump the timeout for instance creation #3468

Merged

skuruppu assigned larkee and unassigned mmmarklu Apr 23, 2020

tmatsuo closed this as completed in #3468 Apr 23, 2020

flaky-bot bot reopened this Apr 24, 2020

tmatsuo closed this as completed Apr 25, 2020

flaky-bot bot reopened this Apr 27, 2020

tmatsuo assigned tmatsuo and unassigned larkee Apr 27, 2020

tmatsuo removed the priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. label Apr 27, 2020

tmatsuo added the priority: p2 Moderately-important priority. Fix may not be included in next release. label Apr 27, 2020

yoshi-automation removed the 🚨 This issue needs some love. label Apr 27, 2020

larkee mentioned this issue May 5, 2020

chore: pin new Cloud Spanner release #3688

Merged

tmatsuo closed this as completed in #3688 May 5, 2020

flaky-bot bot reopened this May 6, 2020

larkee mentioned this issue May 14, 2020

Backup tests are flaky googleapis/python-spanner#83

Closed

tmatsuo closed this as completed May 27, 2020

flaky-bot bot mentioned this issue Jun 8, 2020

spanner.cloud-client.backup_sample_test: test_update_backup failed #4009

Closed

flaky-bot bot mentioned this issue Jun 30, 2020

spanner.cloud-client.backup_sample_test: test_update_backup failed #4213

Closed

This was referenced Jul 16, 2020

spanner.cloud-client.backup_sample_test: test_update_backup failed #4306

Closed

spanner.cloud-client.backup_sample_test: test_update_backup failed #4316

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

spanner.cloud-client.backup_sample_test: test_update_backup failed #3241

spanner.cloud-client.backup_sample_test: test_update_backup failed #3241

flaky-bot bot commented Mar 31, 2020

flaky-bot bot commented Apr 1, 2020

flaky-bot bot commented Apr 1, 2020

flaky-bot bot commented Apr 22, 2020

flaky-bot bot commented Apr 24, 2020

tmatsuo commented Apr 25, 2020

flaky-bot bot commented Apr 27, 2020

tmatsuo commented Apr 27, 2020 •

edited

Loading

tmatsuo commented Apr 27, 2020

flaky-bot bot commented May 6, 2020

tmatsuo commented May 6, 2020

tmatsuo commented May 6, 2020

tmatsuo commented May 7, 2020 •

edited

Loading

tmatsuo commented May 27, 2020

spanner.cloud-client.backup_sample_test: test_update_backup failed #3241

spanner.cloud-client.backup_sample_test: test_update_backup failed #3241

Comments

flaky-bot bot commented Mar 31, 2020

flaky-bot bot commented Apr 1, 2020

flaky-bot bot commented Apr 1, 2020

flaky-bot bot commented Apr 22, 2020

flaky-bot bot commented Apr 24, 2020

tmatsuo commented Apr 25, 2020

flaky-bot bot commented Apr 27, 2020

tmatsuo commented Apr 27, 2020 • edited Loading

tmatsuo commented Apr 27, 2020

flaky-bot bot commented May 6, 2020

tmatsuo commented May 6, 2020

tmatsuo commented May 6, 2020

tmatsuo commented May 7, 2020 • edited Loading

tmatsuo commented May 27, 2020

tmatsuo commented Apr 27, 2020 •

edited

Loading

tmatsuo commented May 7, 2020 •

edited

Loading