java.util.ConcurrentModificationException: null when testing with ~10 concurrent users

[SomeoneToIgnore]

@johannest commented on Thu Jan 11 2018

Bakery load testing with Gatling. When the concurrent user count reaches 5-10 users the exception below starts appearing.

Test scenario

1. Log in with user barista
2. Click new order
3. Fill the form and select 2 products
4. Submit order
5. Navigate to Dashboard

Exception is got quite randomly from different places of the above scenario, e.g. after 3. or when inputing user's name etc.

java.util.ConcurrentModificationException: null
        at java.util.ArrayList.forEach(ArrayList.java:1252) ~[na:1.8.0_45]
        at com.vaadin.server.communication.rpc.MapSyncRpcHandler.flushPendingChangeEvents(MapSyncRpcHandler.java:81) ~[flow-server-1.0.0.alpha13.jar!/:na]
        at com.vaadin.server.communication.ServerRpcHandler.handleInvocations(ServerRpcHandler.java:346) ~[flow-server-1.0.0.alpha13.jar!/:na]
        at com.vaadin.server.communication.ServerRpcHandler.handleRpc(ServerRpcHandler.java:286) ~[flow-server-1.0.0.alpha13.jar!/:na]
        at com.vaadin.server.communication.UidlRequestHandler.synchronizedHandleRequest(UidlRequestHandler.java:88) ~[flow-server-1.0.0.alpha13.jar!/:na]
        at com.vaadin.server.SynchronizedRequestHandler.handleRequest(SynchronizedRequestHandler.java:40) ~[flow-server-1.0.0.alpha13.jar!/:na]
        at com.vaadin.server.VaadinService.handleRequest(VaadinService.java:1517) ~[flow-server-1.0.0.alpha13.jar!/:na]
        at com.vaadin.server.VaadinServlet.service(VaadinServlet.java:351) [flow-server-1.0.0.alpha13.jar!/:na]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar!/:3.1.0]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:728) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
        at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:467) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
        at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:351) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
        at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:311) [tomcat-embed-core-8.5.23.jar!/:8.5.23]
        at org.springframework.web.servlet.mvc.ServletForwardingController.handleRequestInternal(ServletForwardingController.java:141) [spring-webmvc-5.0.2.RELEASE.jar!/:5.0.2.RELEASE]

[SomeoneToIgnore]

@johannest will provide load scripts in the ticket in a short while so that we can try to reproduce it ourselves.

[johannest]

Gatling test script attached:
barista-scalability-test.zip

[SomeoneToIgnore]

Thanks!

[johannest]

Run the test by

1. first downloading Gatling: https://gatling.io/download/ (gatling maven plugin has some issues at the moment)
2. unzip attached archive somewhere (below "/path/to/barista-scalability-test")
3. Navigate to Galing/bin folder
4. Run ./gatling.sh -m -bdf path/to/barista-scalability-test/resources/bodies -sf path/to/barista-scalability-test/scala

[johannest]

Quickly looking code this happens probably because pendingChangeEvents list in MapSyncRpcHandler is modified while flushPendingChangeEvents is run OR flushPendingChangeEvents is called again before it is finished.

[johannest]

As a workaround simply synchronizing the access to the pendingChangeEvents list seems to fix the issue. But synchronizing might cause some other issues such as deadlocks..

[denis-anisimov]

Synchronizing in each way?

We have internal lock on the session which technically should be taken each time when we handle the request unless it's very special request which has to be handled without the lock and it this case it should care about synchronization by itself.

I was pretty sure that all our handlers are executed under the session lock.
Looks like it's not true.
It should be executed under the session lock and it shouldn't cause any additional issues since we have this lock everywhere (technically any sychornization may lead to deadlock if there is a way to run a user code).

[johannest]

I just tried naively adding synchronized block around pendingChangeEvents.add and iteration+clearing of the list. It seemed to "fix" the issue. I realise that this is not valid fix but more like a workaround.

[johannest]

Apparently it is not enough to synchronize the content of flushPendingChangeEvents, thus I suppose that the problem is that pendingChangeEvents.add(changeEventRunnable); is called while the pendingChangeEvents is iterated in the flushPendingChangeEvents method.

[denis-anisimov]

I just tried naively adding synchronized block

OK, thanks, good to know.