ConcurrentModificationException in DefaultStateMachineExecutor

[YusufKizilay]

Hi,

I am observing following exception from state machine on our production environment.
After that exception, state machine is stuck and never process subsequent request.

java.util.ConcurrentModificationException
at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
at java.util.LinkedList$ListItr.next(LinkedList.java:888)
at org.springframework.statemachine.support.DefaultStateMachineExecutor.processDeferList(DefaultStateMachineExecutor.java:406)
at org.springframework.statemachine.support.DefaultStateMachineExecutor.access$200(DefaultStateMachineExecutor.java:61)
at org.springframework.statemachine.support.DefaultStateMachineExecutor$1.run(DefaultStateMachineExecutor.java:274)
at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50)
at org.springframework.statemachine.support.DefaultStateMachineExecutor.scheduleEventQueueProcessing(DefaultStateMachineExecutor.java:292)
at org.springframework.statemachine.support.DefaultStateMachineExecutor.execute(DefaultStateMachineExecutor.java:142)
at org.springframework.statemachine.support.AbstractStateMachine.sendEvent(AbstractStateMachine.java:232)

To reproduce same exception on my local environment , i am sending 1000 concurrent event to same state machine instance and same exception occurs and state machine is stuck. Right now, our production environment does not have a heavy load so i don't think 1000 or much more concurrent access cause our problem.

We are triggerring every event as a defer event which triggers actual event. Can someone help to investigate it?

[jvalkeal]

Which version you're using? Just thinking if a fix needs to be backported.

[YusufKizilay]

Thank you for response Janne. We are using 1.2.3.RELEASE version but i tried same case using latest version which is 2.1.2.RELEASE and i observed same exception.

[jvalkeal]

Ok, I'll try to fix and backport to 2.0.x/1.2.x.

[jvalkeal]

@YusufKizilay Could you check master, if that works for you I can backport this fix next week.

[YusufKizilay]

Hi Janne, i tested the issue using master branch (version is 3.0.0.BUILD-SNAPSHOT) and state machine has behaved strange.

Sometimes when i send n concurrent event to a state machine instance, i observe related state action has triggerred 2xn time even 3xn time. And it is never ending.

But sometimes it is working correctly.

[jvalkeal]

Remember that 3.x is currently going through some internal rewrites for reactive stories. I added a note to README that master may be super unstable.

[jvalkeal]

Master was switched to 3.x last Saturday, current stable branch is 2.1.x.

[YusufKizilay]

If you think infinite loop problem is related to unstable branch, there is no problem right now .We are not getting exception and fix looks work.Please let me know, when you backport the fix to 1.2.x/2.0.x branch and it is ready to use.

[jvalkeal]

Most likely, as I've seen some loops happening with this initial reactive work. Deferred events are slightly broken with new reactive executor which might end up in a loop.

[YusufKizilay]

Hi Janne.
When we can use fix backported version( 1.2.3.RELEASE)?
Have you any guess about this?

[jvalkeal]

I'm doing 1.2.14.RELEASE today.

[jvalkeal]

1.2.14.RELEASE now in spring repos and on its way to central.