Repository navigation
[JENKINS-11256] Maven3 parallel build fails with java.util.ConcurrentModificationException in Jenkins #16148
Description
linux, jenkins 1.433, maven 3.0.3, maven project with maven 3 and -T 8 running on a remote ssh slave
build fails with
— maven-install-plugin:2.3.1:install (default-install) @ xxxxxxxx —
Waiting for Jenkins to finish collecting data
mavenExecutionResult exceptions not empty
message : Internal error: java.util.ConcurrentModificationException
cause : null
Stack trace :
org.apache.maven.InternalErrorException: Internal error: java.util.ConcurrentModificationException
at org.apache.maven.lifecycle.internal.BuilderCommon.handleBuildError(BuilderCommon.java:128)[INFO] Installing xxxxxxxx.jar to xxxxxxxx
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:95)
at org.apache.maven.lifecycle.internal.LifecycleThreadedBuilder$1.call(LifecycleThreadedBuilder.java:167)
at org.apache.maven.lifecycle.internal.LifecycleThreadedBuilder$1.call(LifecycleThreadedBuilder.java:164)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
at java.util.concurrent.FutureTask.run(FutureTask.java:123)[INFO] Compiling 1 source file to xxxxxxxx
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayList.writeObject(ArrayList.java:572)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:592)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
at java.util.ArrayList.writeObject(ArrayList.java:569)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:592)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
at java.util.HashMap.writeObject(HashMap.java:1039)
at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:592)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
at hudson.remoting.UserRequest._serialize(UserRequest.java:155)
at hudson.remoting.UserRequest.serialize(UserRequest.java:164)
at hudson.remoting.UserRequest.(UserRequest.java:62)
at hudson.remoting.Channel.callAsync(Channel.java:708)
at hudson.maven.Maven3Builder$FilterImpl.executeAsync(Maven3Builder.java:229)
at hudson.maven.reporters.MavenFingerprinter.postBuild(MavenFingerprinter.java:96)
at hudson.maven.Maven3Builder$MavenExecutionListener.recordProjectSucceeded(Maven3Builder.java:431)
at hudson.maven.Maven3Builder$MavenExecutionListener.projectSucceeded(Maven3Builder.java:405)
at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire(DefaultExecutionEventCatapult.java:74)
at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire(DefaultExecutionEventCatapult.java:42)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:91)
... 7 more
[INFO]
[INFO] — maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ xxxxxxxx —
channel stopped
Archiviere Artefakte
Finished: FAILURE
Originally reported by kisa, imported from: Maven3 parallel build fails with java.util.ConcurrentModificationException in Jenkins
- assignee:
kutzi
- status: Resolved
- priority: Major
- component(s): core
- resolution: Fixed
- resolved: 2011-11-18T14:20:29+00:00
- votes: 2
- watchers: 8
- imported: 2025-11-24
Raw content of original issue
linux, jenkins 1.433, maven 3.0.3, maven project with maven 3 and -T 8 running on a remote ssh slave build fails with
— maven-install-plugin:2.3.1:install (default-install) @ xxxxxxxx — Waiting for Jenkins to finish collecting data mavenExecutionResult exceptions not empty message : Internal error: java.util.ConcurrentModificationException cause : null Stack trace : org.apache.maven.InternalErrorException: Internal error: java.util.ConcurrentModificationException at org.apache.maven.lifecycle.internal.BuilderCommon.handleBuildError(BuilderCommon.java:128)[INFO] Installing xxxxxxxx.jar to xxxxxxxx
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:95) at org.apache.maven.lifecycle.internal.LifecycleThreadedBuilder$1.call(LifecycleThreadedBuilder.java:167) at org.apache.maven.lifecycle.internal.LifecycleThreadedBuilder$1.call(LifecycleThreadedBuilder.java:164) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269) at java.util.concurrent.FutureTask.run(FutureTask.java:123)[INFO] Compiling 1 source file to xxxxxxxx
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676) at java.lang.Thread.run(Thread.java:595) Caused by: java.util.ConcurrentModificationException at java.util.ArrayList.writeObject(ArrayList.java:572) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:592) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302) at java.util.ArrayList.writeObject(ArrayList.java:569) at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:592) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302) at java.util.HashMap.writeObject(HashMap.java:1039) at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:592) at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:917) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1339) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1375) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1290) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302) at hudson.remoting.UserRequest._serialize(UserRequest.java:155) at hudson.remoting.UserRequest.serialize(UserRequest.java:164) at hudson.remoting.UserRequest.<init>(UserRequest.java:62) at hudson.remoting.Channel.callAsync(Channel.java:708) at hudson.maven.Maven3Builder$FilterImpl.executeAsync(Maven3Builder.java:229) at hudson.maven.reporters.MavenFingerprinter.postBuild(MavenFingerprinter.java:96) at hudson.maven.Maven3Builder$MavenExecutionListener.recordProjectSucceeded(Maven3Builder.java:431) at hudson.maven.Maven3Builder$MavenExecutionListener.projectSucceeded(Maven3Builder.java:405) at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire(DefaultExecutionEventCatapult.java:74) at org.apache.maven.lifecycle.internal.DefaultExecutionEventCatapult.fire(DefaultExecutionEventCatapult.java:42) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:91) ... 7 more [INFO] [INFO] — maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ xxxxxxxx — channel stopped Archiviere Artefakte Finished: FAILURE
- environment:
linux, jenkins 1.433, maven 3.0.3, maven project with maven 3 and -T 8 running on a remote ssh slave
Activity
jenkins-infra-bot commentedon Oct 7, 2011
Raw content of original comment:
Does it happen frequently or only once?
jenkins-infra-bot commentedon Oct 7, 2011
kisa:
Raw content of original comment:
Every 10-15 builds ... bad enough
jenkins-infra-bot commentedon Oct 10, 2011
Raw content of original comment:
All three collections in Maven Fingerprinter need to be synchronized (Collections.synchronized...()) schould be sufficient. This problem could occur at several places.
Also, check wether a given file has alread been digested should be converted into a onestep checkAndSet operation.
I will create a pull request for this.
jenkins-infra-bot commentedon Oct 10, 2011
Raw content of original comment:
Created pauxus@f63bdf3, however it landed in the open pull request concerning Parent POM fingerprinting.
Kutzi, could you look at both of them together or should I split it out again?
jenkins-infra-bot commentedon Oct 10, 2011
Raw content of original comment:
Thanks for the pull request.
Yes, please create a separate pull request for each issue.
Also, I wonder if the 3 maps/sets are really the problem, as they're (as far as I understand it, are per Maven module and therefore only written in a single thread). My suspicion was much more regarding the FingerprintAction which is added and modified in a non-threadsafe fashion. Have you tried if your change fixes the problem?
I think I'd rather put a synchronized block inside the BuildCallable in postBuild()
jenkins-infra-bot commentedon Oct 10, 2011
Raw content of original comment:
Old problem with race conditions - if you need them, they don't show up.
I removed the commit from the pull request - an rechecked again. In my first test, both fingerprinter had very similar hashcodes leading me to the wrong idea that they are reused per module.
As far as I see, the problem occurs during serialization of the BuildCallable itself. Therefore a synchronized block inside BuildCallable should make no difference (Am I overlooking something).
Which leads back to the question: where does ist happen.
The Maven3Builder is shared between modules, it could be the culprit. The stacktrace seems to point to a HashMap containing ArrayLists, with the ArrayList being modified. The ony qualifiying map that I have found would be Maven3Builder.reporters (which would match the stack trace)
However, I did not found anyone modifying one of the valuelists.
Changing MavenModule.createReporters to return a synchronized List should solve the problem, however, I don't like not knowing where the problem actually arises.
I will further look into this.
UPDATE:
Its an ArrayList in an ArrayList in the reporters map could be some plugin.
jenkins-infra-bot commentedon Oct 15, 2011
Raw content of original comment:
According to Olivier this might be a bug in Maven 3.0.3:
http://jenkins.361315.n4.nabble.com/Maven2-3-Parallel-Builds-tp3896403p3904271.html
jenkins-infra-bot commentedon Nov 7, 2011
Raw content of original comment:
Short update:
I could not reproduce the problem using a test installation, I will try another test using the exact plugin configuration of the actual system.
jenkins-infra-bot commentedon Nov 17, 2011
Raw content of original comment:
Found the culprit (no plugin):
hudson.maven.reporters.SurefireArchiver.result is a TestResult object containing an ArrayList (suites). This list is updated whenever a module containing tests completes. Since The Async invoke serializes the Maven3Builder and therefore all its reporters (along with the SurefireArchivers). If the fingerprinter runs during the update of the test result, this error occurs.
Solution: Make the field hudson.tasks.junit.TestResult.suites a synchronized list (Collections.synchronizedList)
jenkins-infra-bot commentedon Nov 17, 2011
Raw content of original comment:
@kutzi: My development environment seems broken. Could you do that change?
Thx
jenkins-infra-bot commentedon Nov 17, 2011
Raw content of original comment:
Ah, forget what I previously wrote here, I missed an important part of your comments.
Did you manage to reproduce it reliably? If so, do you have a test case?
jenkins-infra-bot commentedon Nov 17, 2011
Raw content of original comment:
While it might make sense to make TestResult thread-safe, I'd also ask if it's actually needed that SurefireArchiver#result is serialized.
Maybe that could be changed, too?
jenkins-infra-bot commentedon Nov 18, 2011
Raw content of original comment:
Code changed in jenkins
User: Christoph Kutzinski
Path:
maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
http://jenkins-ci.org/commit/jenkins/1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4
Log:
[FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter
Compare: 796918d...1476b2e
jenkins-infra-bot commentedon Nov 18, 2011
Raw content of original comment:
I've removed the serialization of 'result' and local tests indicate that this fixes the issue.
As a positive side effect, Jenkins has to serialize less stuff on each remote call, which should reduce the Maven job overhead a bit
jenkins-infra-bot commentedon Nov 18, 2011
Raw content of original comment:
Code changed in jenkins
User: Christoph Kutzinski
Path:
changelog.html
http://jenkins-ci.org/commit/jenkins/b835fb15497cf31a5069804a9920a747053be3c4
Log:
Changelog entry for
JENKINS-11256jenkins-infra-bot commentedon Nov 18, 2011
Raw content of original comment:
Christoph Kutzinski : b835fb15497cf31a5069804a9920a747053be3c4 Files :
Integrated in
jenkins_main_trunk #1303
[FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter
Changelog entry for
JENKINS-11256Christoph Kutzinski : 1476b2ee446f27b0c4ce85c16c5f984ef84cc8a4
Files :
Christoph Kutzinski : b835fb15497cf31a5069804a9920a747053be3c4
Files :
jenkins-infra-bot commentedon Dec 15, 2011
swolk:
Raw content of original comment:
When will this make it into an LTS build? I really need this fix. Thanks
jenkins-infra-bot commentedon Dec 16, 2011
Raw content of original comment:
Code changed in jenkins
User: Christoph Kutzinski
Path:
maven-plugin/src/main/java/hudson/maven/reporters/SurefireArchiver.java
http://jenkins-ci.org/commit/jenkins/46629db8d493f67c5b8ce1c14465b3c6a117e847
Log:
[FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter(cherry picked from commit 1476b2e)
jenkins-infra-bot commentedon Dec 16, 2011
Raw content of original comment:
Code changed in jenkins
User: Christoph Kutzinski
Path:
changelog.html
http://jenkins-ci.org/commit/jenkins/b39a310c5165234dab6d7acd3ed82bd399d255eb
Log:
Changelog entry for
JENKINS-11256(cherry picked from commit b835fb1)Conflicts:
changelog.html
jenkins-infra-bot commentedon Oct 4, 2013
Raw content of original comment:
Code changed in jenkins
User: Christoph Kutzinski
Path:
src/main/java/hudson/maven/reporters/SurefireArchiver.java
http://jenkins-ci.org/commit/maven-plugin/00f1af0b4e71f8c691e36a1bb741eeff7bd512ee
Log:
[FIXED JENKINS-11256] SurefireArchiver#result doesn't need to be serialized on each remote call. Fixes ConcurrentModificationExceptions in MavenFingerprinter
Originally-Committed-As: 1476b2e