Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

spanner.it.ITBackupTest: testBackups failed #542

Closed
flaky-bot bot opened this issue Oct 23, 2020 · 4 comments · Fixed by #557
Closed

spanner.it.ITBackupTest: testBackups failed #542

flaky-bot bot opened this issue Oct 23, 2020 · 4 comments · Fixed by #557
Assignees
Labels
api: spanner priority: p0 🚨 type: bug

Comments

@flaky-bot
Copy link

@flaky-bot flaky-bot bot commented Oct 23, 2020

Note: #481 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.


commit: eddd6ad
buildURL: Build Status, Sponge
status: failed

Test output
com.google.cloud.spanner.SpannerException: DEADLINE_EXCEEDED: Backup1 still not finished. Test is giving up waiting for it.
	at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:236)
	at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:56)
	at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:51)
	at com.google.cloud.spanner.it.ITBackupTest.testBackups(ITBackupTest.java:293)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeLazy(JUnitCoreWrapper.java:119)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:87)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
	at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
@flaky-bot flaky-bot bot added buildcop: issue priority: p1 type: bug labels Oct 23, 2020
@product-auto-label product-auto-label bot added the api: spanner label Oct 23, 2020
@flaky-bot
Copy link
Author

@flaky-bot flaky-bot bot commented Oct 25, 2020

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.


When run at the same commit (eddd6ad), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).

@lesv lesv added priority: p0 and removed priority: p1 labels Oct 26, 2020
@lesv
Copy link
Contributor

@lesv lesv commented Oct 26, 2020

Looking in Java-docs-samples-testing, I'm finding 9 spanner instances w/ backups in Asia we are now out of quota. Please fix the code so you actually delete the instance.

@lesv
Copy link
Contributor

@lesv lesv commented Oct 26, 2020

Note - you have to delete the backup before you delete the instance. (at least in the console)

@yoshi-automation yoshi-automation added the 🚨 label Oct 26, 2020
@hengfengli
Copy link
Contributor

@hengfengli hengfengli commented Oct 27, 2020

@olavloite It seems that we have uncleaned instances for backups.

olavloite added a commit that referenced this issue Oct 27, 2020
An instance can only be deleted once all backups on the instance have been
deleted. The integration test environment should therefore first delete all
backups on an owned instance before trying to delete the instance itself.

Fixes #542
thiagotnunes pushed a commit that referenced this issue Oct 27, 2020
* fix: always delete all backups from an owned test instance

An instance can only be deleted once all backups on the instance have been
deleted. The integration test environment should therefore first delete all
backups on an owned instance before trying to delete the instance itself.

Fixes #542

* fix: do not try to list backups on emulator
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: spanner priority: p0 🚨 type: bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants