Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky test: ITBigQueryStorageTest.testIngestionTimePartitionedTable #305

Closed
stephaniewang526 opened this issue May 28, 2020 · 3 comments
Closed

Flaky test: ITBigQueryStorageTest.testIngestionTimePartitionedTable #305

stephaniewang526 opened this issue May 28, 2020 · 3 comments
Assignees
@kmjung
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@stephaniewang526
Copy link
Contributor

https://source.cloud.google.com/results/invocations/bd8e53cc-4f88-477a-bc07-88a27406cbca/targets/github%2Fjava-bigquerystorage%2Fgoogle-cloud-bigquerystorage%2Ftarget%2Ffailsafe-reports%2Fintegration%2Fcom.google.cloud.bigquery.storage.v1beta1.it.ITBigQueryStorageLongRunningTest/tests

com.google.cloud.bigquery.BigQueryException: com.google.api.gax.retrying.PollException: total timeout or maximum number of attempts exceeded; current settings: RetrySettings{totalTimeout=PT1M, initialRetryDelay=PT1S, retryDelayMultiplier=2.0, maxRetryDelay=PT1M, maxAttempts=0, jittered=true, initialRpcTimeout=PT0S, rpcTimeoutMultiplier=1.0, maxRpcTimeout=PT0S}
	at com.google.cloud.bigquery.BigQueryException.translateAndThrow(BigQueryException.java:105)
	at com.google.cloud.bigquery.Job.waitForQueryResults(Job.java:341)
	at com.google.cloud.bigquery.Job.waitFor(Job.java:240)
	at com.google.cloud.bigquery.storage.v1.it.ITBigQueryStorageTest.RunQueryJobAndExpectSuccess(ITBigQueryStorageTest.java:953)
	at com.google.cloud.bigquery.storage.v1.it.ITBigQueryStorageTest.RunQueryAppendJobAndExpectSuccess(ITBigQueryStorageTest.java:933)
	at com.google.cloud.bigquery.storage.v1.it.ITBigQueryStorageTest.testIngestionTimePartitionedTable(ITBigQueryStorageTest.java:453)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
	at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
Caused by: com.google.api.gax.retrying.PollException: total timeout or maximum number of attempts exceeded; current settings: RetrySettings{totalTimeout=PT1M, initialRetryDelay=PT1S, retryDelayMultiplier=2.0, maxRetryDelay=PT1M, maxAttempts=0, jittered=true, initialRpcTimeout=PT0S, rpcTimeoutMultiplier=1.0, maxRpcTimeout=PT0S}
	at com.google.api.gax.retrying.ExponentialPollAlgorithm.shouldRetry(ExponentialPollAlgorithm.java:70)
	at com.google.api.gax.retrying.RetryAlgorithm.shouldRetry(RetryAlgorithm.java:116)
	at com.google.api.gax.retrying.BasicRetryingFuture.handleAttempt(BasicRetryingFuture.java:167)
	at com.google.api.gax.retrying.BasicRetryingFuture.setAttemptFuture(BasicRetryingFuture.java:94)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:106)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
	at com.google.cloud.RetryHelper.poll(RetryHelper.java:64)
	at com.google.cloud.bigquery.Job.waitForQueryResults(Job.java:325)
	... 45 more
@stephaniewang526 stephaniewang526 mentioned this issue May 28, 2020
Merged
@stephaniewang526 stephaniewang526 changed the title IT failure: ITBigQueryStorageTest.testIngestionTimePartitionedTable Flaky test: ITBigQueryStorageTest.testIngestionTimePartitionedTable May 28, 2020
@kmjung
Copy link
Contributor

kmjung commented May 28, 2020

This isn't actually a storage API failure -- it's a timeout while waiting for a query to execute:

com.google.cloud.bigquery.BigQueryException: com.google.api.gax.retrying.PollException: total timeout or maximum number of attempts exceeded; current settings: RetrySettings{totalTimeout=PT1M, initialRetryDelay=PT1S, retryDelayMultiplier=2.0, maxRetryDelay=PT1M, maxAttempts=0, jittered=true, initialRpcTimeout=PT0S, rpcTimeoutMultiplier=1.0, maxRpcTimeout=PT0S}
	at com.google.cloud.bigquery.BigQueryException.translateAndThrow(BigQueryException.java:105)
	at com.google.cloud.bigquery.Job.waitForQueryResults(Job.java:341)
	at com.google.cloud.bigquery.Job.waitFor(Job.java:240)

Beyond that, it's a little bit hard to say more -- these logs don't contain a BigQuery job ID or anything else explicit to go on. Could we add this logging?

@stephaniewang526
Copy link
Contributor Author

Yes - i think we could add more logging

@yoshi-automation yoshi-automation added the triage me I really want to be triaged. label May 29, 2020
@stephaniewang526 stephaniewang526 added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. and removed triage me I really want to be triaged. labels May 29, 2020
@stephaniewang526
Copy link
Contributor Author

@kmjung Happening again today on this PR #316

https://source.cloud.google.com/results/invocations/21e033ab-0637-4fad-8f66-476773f0915b/targets/github%2Fjava-bigquerystorage%2Fgoogle-cloud-bigquerystorage%2Ftarget%2Ffailsafe-reports%2Fintegration%2Fcom.google.cloud.bigquery.storage.v1beta2.it.ITBigQueryStorageLongRunningTest/tests

@yoshi-automation yoshi-automation added triage me I really want to be triaged. 🚨 This issue needs some love. labels May 29, 2020
@chingor13 chingor13 added the priority: p2 Moderately-important priority. Fix may not be included in next release. label Jun 3, 2020
@yoshi-automation yoshi-automation removed triage me I really want to be triaged. 🚨 This issue needs some love. labels Jun 3, 2020
shubhwip pushed a commit to shubhwip/java-bigquerystorage that referenced this issue Oct 7, 2023
@renovate-bot
deps: update dependency com.google.cloud:google-cloud-shared-dependen…
79245ae 
…cies to v0.16.0 (googleapis#305)
shubhwip pushed a commit to shubhwip/java-bigquerystorage that referenced this issue Oct 7, 2023
@release-please
chore: release 0.122.3 (googleapis#308)
0e7901c 
🤖 I have created a release \*beep\* \*boop\* 
---
### [0.122.3](https://www.github.com/googleapis/java-storage-nio/compare/v0.122.2...v0.122.3) (2020-12-03)


### Dependencies

* update dependency com.google.cloud:google-cloud-shared-dependencies to v0.16.0 ([googleapis#305](https://www.github.com/googleapis/java-storage-nio/issues/305)) ([79245ae](https://www.github.com/googleapis/java-storage-nio/commit/79245ae5ded370d310f31939aa36572563437bb3))
---


This PR was generated with [Release Please](https://github.com/googleapis/release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kmjung kmjung

Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@chingor13 @stephaniewang526 @kmjung @yoshi-automation