Siddhi App 56d572bd-caaa-41c5-928a-26813ae8bcd2 not stabilized for snapshot/restore, Active thread count is 1

[faf-xff]

Description:
not stabilized for snapshot/restore, Active thread count is 1
Affected Siddhi Version:
siddhi-core:5.1.7
OS, DB, other environment details and versions:
I use the jar package after the integration of Flink and siddhi (flink-siddhi.jar) to run in Hadoop.
https://github.com/haoch/flink-siddhi
Steps to reproduce:
exception information:
io.siddhi.core.exception.SiddhiAppRuntimeException: Siddhi App 47b2c172-51e7-45ff-b471-1ac16de23e5f not stabilized for snapshot/restore, Active thread count is 1
at io.siddhi.core.util.snapshot.SnapshotService.waitForSystemStabilization(SnapshotService.java:768)
at io.siddhi.core.util.snapshot.SnapshotService.fullSnapshot(SnapshotService.java:100)
at io.siddhi.core.SiddhiAppRuntimeImpl.snapshot(SiddhiAppRuntimeImpl.java:674)
at org.apache.flink.streaming.siddhi.operator.AbstractSiddhiOperator.checkpointSiddhiRuntimeState(AbstractSiddhiOperator.java:377)
at org.apache.flink.streaming.siddhi.operator.AbstractSiddhiOperator.processElement(AbstractSiddhiOperator.java:220)
at org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:202)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:105)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
at java.lang.Thread.run(Thread.java:745)
Related Issues:
What causes this problem and how to solve it? Thank you

[pcnfernando]

This exception is thrown during Siddhi snapshot/restore process.
As per the exception context, it seems one thread from Siddhi thread pool, is in RUNNING state without going to BLOCKED state after completing its task.
I suspect It might be caught in a loop or somewhere interrupt mechanism is not correctly implemented within a custom extension.

Are you using any Siddhi custom extensions ?

[faf-xff]

Are you using any Siddhi custom extensions ?

No, I used string extension and regex extension, but I found that if I used the window.time()and window.timeBatch(), I would report an exception. If I didn't use the window.time()and window.timeBatch(), there would be no exception
siddhi-execution-string:5.0.7
siddhi-execution-regex:5.0.5

[pcnfernando]

Can you please provide few thread dumps at regular intervals(eg: 10s intervals) when this scenario is experienced

[faf-xff]

OK, I'll try. Thanks

[pcnfernando]

Are you using any Siddhi custom extensions ?

No, I used string extension and regex extension, but I found that if I used the window.time()and window.timeBatch(), I would report an exception. If I didn't use the window.time()and window.timeBatch(), there would be no exception
siddhi-execution-string:5.0.7
siddhi-execution-regex:5.0.5

Furthermore, siddhi-execution-string and siddhi-execution-regex are stateless processes. Hence, the Siddhi snapshot/restore tasks will not be triggered with the said extensions. Whereas, window operations are stateful processes which triggers Siddhi snapshot/restore tasks.

[pcnfernando]

Can you please verify whether you tried with Siddhi v5.1.7 or v5.1.2?
Within https://github.com/haoch/flink-siddhi the Siddhi version used is v5.1.2

[faf-xff]

Can you please verify whether you tried with Siddhi v5.1.7 or v5.1.2?
Within https://github.com/haoch/flink-siddhi the Siddhi version used is v5.1.2

Flink-siddhi.jar requires user to load the siddhi.jar package. My version is 5.1.7

[pcnfernando]

@guanyufen123 Were you able to capture the thread dumps as I requested?

[faf-xff]

ok,thanks