Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SamzaStoreStateInternalsTest is flaky #20977

Closed
damccorm opened this issue Jun 4, 2022 · 1 comment
Closed

SamzaStoreStateInternalsTest is flaky #20977

damccorm opened this issue Jun 4, 2022 · 1 comment
Assignees
@kennknowles
Labels
bug done & done Issue has been reviewed after it was closed for verification, followups, etc. failing test flake java P1 runners samza
Milestone
2.44.0 Release

Comments

@damccorm
Copy link
Contributor

damccorm commented Jun 4, 2022

Happened in precommits: https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/17290/testReport/junit/org.apache.beam.runners.samza.runtime/


Error Message
org.apache.beam.sdk.Pipeline$PipelineExecutionException: org.apache.samza.SamzaException:
Exception when restoring 
Stacktrace
org.apache.beam.sdk.Pipeline$PipelineExecutionException: org.apache.samza.SamzaException:
Exception when restoring 
	at org.apache.beam.runners.samza.SamzaPipelineResult.getStateInfo(SamzaPipelineResult.java:129)
	at
org.apache.beam.runners.samza.SamzaPipelineResult.waitUntilFinish(SamzaPipelineResult.java:92)
	at
org.apache.beam.runners.samza.SamzaPipelineResult.waitUntilFinish(SamzaPipelineResult.java:108)
	at
org.apache.beam.runners.samza.TestSamzaRunner.run(TestSamzaRunner.java:78)
	at org.apache.beam.sdk.Pipeline.run(Pipeline.java:322)
	at
org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:398)
	at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:334)
	at
org.apache.beam.runners.samza.runtime.SamzaStoreStateInternalsTest.testSetStateIterator(SamzaStoreStateInternalsTest.java:192)
	at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:322)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at
org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at
org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
	at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
	at
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
	at
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
	at
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at
com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
	at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
	at
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
	at
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
	at
java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.samza.SamzaException: Exception when restoring

	at org.apache.samza.storage.ContainerStorageManager.restoreStores(ContainerStorageManager.java:695)
	at
org.apache.samza.storage.ContainerStorageManager.start(ContainerStorageManager.java:653)
	at org.apache.samza.container.SamzaContainer.startStores(SamzaContainer.scala:966)
	at
org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:764)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more
Caused
by: java.util.concurrent.ExecutionException: org.apache.samza.SamzaException: Error opening RocksDB
store beamStore at location /tmp/beam-samza-test/beamStore/Partition_0
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at
java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.samza.storage.ContainerStorageManager.restoreStores(ContainerStorageManager.java:687)
	...
8 more
Caused by: org.apache.samza.SamzaException: Error opening RocksDB store beamStore at location
/tmp/beam-samza-test/beamStore/Partition_0
	at org.apache.samza.storage.kv.RocksDbKeyValueStore$.openDB(RocksDbKeyValueStore.scala:107)
	at
org.apache.samza.storage.kv.RocksDbKeyValueStore.org$apache$samza$storage$kv$RocksDbKeyValueStore$$db$lzycompute(RocksDbKeyValueStore.scala:125)
	at
org.apache.samza.storage.kv.RocksDbKeyValueStore.org$apache$samza$storage$kv$RocksDbKeyValueStore$$db(RocksDbKeyValueStore.scala:125)
	at
org.apache.samza.storage.kv.RocksDbKeyValueStore$$anonfun$putAll$1.apply$mcJ$sp(RocksDbKeyValueStore.scala:195)
	at
org.apache.samza.storage.kv.RocksDbKeyValueStore$$anonfun$putAll$1.apply(RocksDbKeyValueStore.scala:176)
	at
org.apache.samza.storage.kv.RocksDbKeyValueStore$$anonfun$putAll$1.apply(RocksDbKeyValueStore.scala:176)
	at
org.apache.samza.storage.kv.RocksDbKeyValueStore.org$apache$samza$storage$kv$RocksDbKeyValueStore$$ifOpen(RocksDbKeyValueStore.scala:275)
	at
org.apache.samza.storage.kv.RocksDbKeyValueStore.putAll(RocksDbKeyValueStore.scala:176)
	at org.apache.samza.storage.kv.SerializedKeyValueStore.putAll(SerializedKeyValueStore.scala:89)
	at
org.apache.samza.storage.kv.CachedStore.org$apache$samza$storage$kv$CachedStore$$putAllDirtyEntries(CachedStore.scala:218)
	at
org.apache.samza.storage.kv.CachedStore.flush(CachedStore.scala:198)
	at org.apache.samza.storage.kv.NullSafeKeyValueStore.flush(NullSafeKeyValueStore.scala:85)
	at
org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$flush$1.apply$mcV$sp(KeyValueStorageEngine.scala:214)
	at
org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$flush$1.apply(KeyValueStorageEngine.scala:211)
	at
org.apache.samza.storage.kv.KeyValueStorageEngine$$anonfun$flush$1.apply(KeyValueStorageEngine.scala:211)
	at
org.apache.samza.util.TimerUtil$class.updateTimer(TimerUtil.scala:37)
	at org.apache.samza.storage.kv.KeyValueStorageEngine.updateTimer(KeyValueStorageEngine.scala:39)
	at
org.apache.samza.storage.kv.KeyValueStorageEngine.flush(KeyValueStorageEngine.scala:211)
	at org.apache.samza.storage.kv.KeyValueStorageEngine.close(KeyValueStorageEngine.scala:235)
	at
org.apache.samza.storage.kv.KeyValueStorageEngine.stop(KeyValueStorageEngine.scala:229)
	at org.apache.samza.storage.TransactionalStateTaskRestoreManager.lambda$stopPersistentStores$0(TransactionalStateTaskRestoreManager.java:148)
	at
java.util.HashMap.forEach(HashMap.java:1289)
	at org.apache.samza.storage.TransactionalStateTaskRestoreManager.stopPersistentStores(TransactionalStateTaskRestoreManager.java:146)
	at
org.apache.samza.storage.ContainerStorageManager$TaskRestoreCallable.call(ContainerStorageManager.java:952)
	at
org.apache.samza.storage.ContainerStorageManager$TaskRestoreCallable.call(ContainerStorageManager.java:922)
	...
4 more
Caused by: org.rocksdb.RocksDBException: lock hold by current process, acquire time 1618957171
acquiring thread 140263348766464: /tmp/beam-samza-test/beamStore/Partition_0/LOCK: No locks available
	at
org.rocksdb.RocksDB.open(Native Method)
	at org.rocksdb.RocksDB.open(RocksDB.java:251)
	at org.apache.samza.storage.kv.RocksDbKeyValueStore$.openDB(RocksDbKeyValueStore.scala:75)
	...
28 more

Imported from Jira BEAM-12200. Original Jira may contain additional context.
Reported by: tvalentyn.
@kennknowles kennknowles self-assigned this Oct 21, 2022
@kennknowles
Copy link
Member

Didn't repro, and isn't disabled or flaking now.

(set +e
for i in `seq 1 1000` ; do
  echo "RUN $i"
  ./gradlew --quiet :runners:samza:cleanTest
  ./gradlew --quiet :runners:samza:test \
      --tests 'SamzaStoreStateInternalsTest'
done
)

@github-actions github-actions bot added this to the 2.44.0 Release milestone Oct 21, 2022
@apilloud apilloud added the done & done Issue has been reviewed after it was closed for verification, followups, etc. label Oct 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kennknowles kennknowles

Labels
bug done & done Issue has been reviewed after it was closed for verification, followups, etc. failing test flake java P1 runners samza
Projects
None yet
Milestone
2.44.0 Release
Development

No branches or pull requests
3 participants
@kennknowles @apilloud @damccorm