Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

spanner: unflake integration tests #1719

Closed
tritone opened this issue Jan 7, 2020 · 5 comments
Closed

spanner: unflake integration tests #1719

tritone opened this issue Jan 7, 2020 · 5 comments
Assignees
@olavloite
Labels
api: spanner Issues related to the Spanner API. type: process A process-related concern. May include testing, release, or the like.

Comments

@tritone
Copy link
Contributor

tritone commented Jan 7, 2020

Flakes for multiple spanner integration tests over the past day:

TestIntegration_DML: https://sponge.corp.google.com/target?id=88c22d5e-bb5c-4965-acb8-021c8346d4cc&target=cloud-devrel/client-libraries/go/google-cloud-go/continuous/go113&searchFor=&show=ALL&sortBy=STATUS

TestBatchDML_TwoStatements: https://sponge.corp.google.com/target?id=d7c51dd9-ba08-47f6-9305-2ad6438e0fc0&target=cloud-devrel/client-libraries/go/google-cloud-go/continuous/go112&searchFor=&show=ALL&sortBy=STATUS

TestIntegration_QueryExpressions: https://sponge.corp.google.com/target?id=cf295096-310f-4d5d-b806-a07a5bab09dd&target=cloud-devrel/client-libraries/go/google-cloud-go/continuous/go113&searchFor=&show=ALL&sortBy=STATUS

multiple tests failed: https://sponge.corp.google.com/target?id=6106295f-6a4b-403f-b637-091ea191529b&target=cloud-devrel/client-libraries/go/google-cloud-go/continuous/go113&searchFor=&show=ALL&sortBy=STATUS

Please fix these (and/or disable flaky tests for the time being if there's not a quick fix).
@tritone tritone added api: spanner Issues related to the Spanner API. type: process A process-related concern. May include testing, release, or the like. labels Jan 7, 2020
@tritone tritone changed the title Unflake Spanner integration tests spanner: unflake integration tests Jan 7, 2020
@noahdietz
Copy link
Contributor

I encountered some flakiness on

--- FAIL: TestBatchCreateSessions_WithTimeout (0.04s)
    sessionclient_test.go:311: Error code mismatch
        got: Unknown
        want: DeadlineExceeded

https://sponge.corp.google.com/target?tab=Test+Log&sortBy=STATUS&show=ALL&id=77702e48-fe76-4111-bfef-1642fc04f09d&target=cloud-devrel/client-libraries/go/google-cloud-go/presubmit/go112&searchFor=

@olavloite olavloite mentioned this issue Jan 8, 2020
Closed
@olavloite
Copy link
Contributor

The first two flakes will be solved when #1720 is solved.

@olavloite
Copy link
Contributor

The flake in #1719 (comment) has been fixed by https://code-review.googlesource.com/c/gocloud/+/50231

gopherbot pushed a commit that referenced this issue Jan 10, 2020
@olavloite
spanner: session client should return Spanner errors
f858c6c 
The session client would return non-Spanner errors to the session
consumer when a context error or a gRPC error would occur.

Updates #1719.

Change-Id: Ife5b6cc503609404a9b8abed290fdb1932b29a56
Reviewed-on: https://code-review.googlesource.com/c/gocloud/+/50231
Reviewed-by: kokoro <noreply+kokoro@google.com>
Reviewed-by: Shanika Kuruppu <skuruppu@google.com>
@olavloite
Copy link
Contributor

The flaky failures in TestIntegration_QueryExpressions and also in the multiple failures are the following:

--- FAIL: TestIntegration_QueryExpressions (4.41s)
    integration_test.go:2492: cannot create testing DB projects/dulcet-port-762/instances/gotest-1578366254064604995-01/databases/gotest_1578366254064516105_12: rpc error: code = NotFound desc = Operation not found: projects/dulcet-port-762/instances/gotest-1578366254064604995-01/databases/gotest_1578366254064516105_12/operations/_auto_1578366483090205

This error however comes directly from the generated method databaseAdmin.CreateDatabase and is not something that we can do anything about in the client library. It is also a rather unexpected error, as it indicates that the CreateDatabase call failed with a NOT_FOUND error for the long-running operation that monitors the creation of the database. I'm closing this for now, as all the other failures that were actionable for the client library have been fixed. Please feel free to re-open the issue if these failures continue to occur. Then we have to do a more thorough investigation into why that generated method (sometimes) fails with this error.

@tritone
Copy link
Contributor Author

tritone commented Jan 28, 2020

Thanks for taking care of this, and for the detailed description of your investigation!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@olavloite olavloite

Labels
api: spanner Issues related to the Spanner API. type: process A process-related concern. May include testing, release, or the like.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@olavloite @tritone @noahdietz