[MNG-6843] - Parallel build fails due to missing JAR artifacts in compilePath #310

kodeva · 2020-01-06T10:51:55Z

Description

The MavenProject objects are shared among all build threads via MavenSession - member org.apache.maven.execution.MavenSession#projects.
As all the MavenSession objects are created by cloning an initial MavenSession object, I have added cloning of projects into the MavenSession#clone method.
With this change each build has its own copy of MavenProject objects and this way unintention changes of the objects from different build threads is avoided.

Following this checklist to help us incorporate your
contribution quickly and easily:

Make sure there is a JIRA issue filed
for the change (usually before you start working on it). Trivial changes like typos do not
require a JIRA issue. Your pull request should address just this issue, without
pulling in other changes.
Each commit in the pull request should have a meaningful subject line and body.
Format the pull request title like [MNG-XXX] - Fixes bug in ApproximateQuantiles,
where you replace MNG-XXX with the appropriate JIRA issue. Best practice
is to use the JIRA issue title in the pull request title and in the first line of the
commit message.
Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
Run mvn clean verify to make sure basic checks pass. A more thorough check will
be performed on your pull request automatically.
You have run the Core IT successfully.

If your pull request is about ~20 lines of code you don't need to sign an
Individual Contributor License Agreement if you are unsure
please ask on the developers list.

To make clear that you license your contribution under
the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

I hereby declare this contribution to be licenced under the Apache License Version 2.0, January 2004
In any other case, please file an Apache Individual Contributor License Agreement.

Each build session will then have its own copy of MavenProject objects and cannot be influenced, for example, get determining compile classpath by other running build threads.

rfscholte · 2020-01-06T21:15:06Z

I wonder if this is the correct fix. This is pretty old code I guess, but I'd like to know why the cloning is required. We've work quite a lot on performance improvements and minimizing the memoryfootprint lately, cloning would have a counter effect.

kodeva · 2020-01-07T08:16:53Z

The usage of cloned MavenSession objects with shallow copy of MavenProject objects is not thread-safe when it comes to parallel builds because the MavenProjects are being modified from multiple threads. See the https://issues.apache.org/jira/browse/MNG-6843 issue.

I don't have big insight in maven code and thus yep, this change fixes something while it might be breaking something different not covered by automated tests. And, most probably, there's another better fix for the issue.

I'll try to create a project that simulates the issue deterministically and that can be publicly shared.

famod · 2020-04-12T18:43:30Z

I am also affected by this rather severe bug (see Jira issue MNG-6843).

I had a look at the code involved (before this PR patch) and I do not understand why MojoExecutor.ensureDependenciesAreResolved() is setting the ArtifactFilter for multiple projects?!

        ArtifactFilter artifactFilter = getArtifactFilter( mojoDescriptor );
        List<MavenProject> projectsToResolve =
            LifecycleDependencyResolver.getProjects( session.getCurrentProject(), session,
                                                     mojoDescriptor.isAggregator() );
        for ( MavenProject projectToResolve : projectsToResolve )
        {
            projectToResolve.setArtifactFilter( artifactFilter );
        }

The next step after ensureDependenciesAreResolved() is the actual execution of the respective mojo via org.apache.maven.plugin.BuildPluginManager.executeMojo(MavenSession, MojoExecution).
The default implementation (DefaultBuildPluginManager) is only operating on a single MavenProject, not multiple ones.

Am I missing something, @rfscholte?

rfscholte · 2020-04-13T07:44:07Z

This will require some good analysis, maybe an annotate/blame on the sourcecode give a hint.

famod · 2020-04-13T11:32:48Z

This was added 10 years ago: 0ce5541#diff-2d99239b7db610b69d251224cdf93dc6R269-R277
Ticket: MNG-4795 "[regression] Dependencies in forked reactor projects are not resolved when aggregator bound to lifecycle forks"

famod · 2020-04-13T13:41:27Z

Three ITs fail when just setting the ArtifactFilter for dependencyContext.getProject():

[ERROR] Failures:
[ERROR]   MavenITmng4320AggregatorAndDependenciesTest>AbstractMavenIntegrationTestCase.runTest:222->testit:65 [test-classes, classes]
[ERROR]   MavenITmng4331DependencyCollectionTest>AbstractMavenIntegrationTestCase.runTest:222->testitCliAggregator:86 []
[ERROR]   MavenITmng5135AggregatorDepResolutionModuleExtensionTest>AbstractMavenIntegrationTestCase.runTest:222->testit:63 [test-classes, classes]
[INFO]
[ERROR] Tests run: 826, Failures: 3, Errors: 0, Skipped: 0

famod · 2020-04-13T16:46:50Z

This was added 10 years ago: 0ce5541#diff-2d99239b7db610b69d251224cdf93dc6R269-R277
Ticket: MNG-4795 "[regression] Dependencies in forked reactor projects are not resolved when aggregator bound to lifecycle forks"

I need to correct myself: This is the first commit that added the filter handling: d0d3f58
Ticket: MNG-4526 [regression] Shade plugin includes too much (relative to 2.2.1)

famod · 2020-04-13T20:20:59Z

This is interesting and a bit frustating: neither this PR nor another approach I implemented locally (with ThreadLocal) are able to fix this problem for me.
On the other hand it does not really surprise me, given all the mutating methods in MavenProject (set*) that seem like a bit of a mess in combination with concurrent access...

@kodeva Can you verify that this PR fixes your problem?

fkalinski · 2020-04-21T11:48:10Z

@famod I was also going to test the ThreadLocal approach, as it seems as an easiest workaround.

I have tried an approach with ThreadLocal around MavenProject.artifactMap,artifactFilter and artifacts. Seemed to work for me, but didn't test it extensively, and it was a few months ago.

Have you tried the same?

famod · 2020-04-21T18:44:14Z

@fkalinski This was my approach: famod/maven@master...MNG-6843-parallel-classpath

fkalinski · 2020-04-21T19:24:13Z

I've done actually the same change and it works for me.
The only difference, is not factoring the ThreadlLocals into a common class, but plainly using the three of them on the same fields (fkalinski@b38e3c2)

Before the change, parallel builds were failing around 30% of the time, on the 64-module project I work on. Due to many inter-dependencies, maximum parallelism than can be achieved on the project is around 4. When build fails. it fails fast, as the initial modules that are built are of a small size.

After the change applied again today, I haven't seen a failed build, yet.

fkalinski · 2020-04-22T13:36:02Z

As I have checked, the havoc is caused by ensureDependenciesAreResolved setting the artifactFilter to null on all the projects when executing "effective-pom" aggregate mojo.
All the other mojos are affecting only the one module for which the mojo is executed.

As it's an aggregator, it writes effective pom for each projects. During that it accesses project.getDependencies(), therefore clearing the artifact filters makes some sense, despite that doing that by modifying MavenProject objects is really ugly.

Therefore, the problem lies in handling of aggregator mojos, and it's definitely a Maven bug to be fixed. The workaround with ThreadLocals works for me, and it actually should be due to nature of the problem. It would be interesting to see what actaully happens for you, @famod .

And specifically in the project I am working on, "maven-help-plugin", with it's "effective-pom" mojo has been configured on the parent project level, thus, clearing the artifact list on execution of all the modules. After moving it to the top pom project parallel build finally works for me!

kodeva · 2020-04-24T07:07:08Z

This is interesting and a bit frustating: neither this PR nor another approach I implemented locally (with ThreadLocal) are able to fix this problem for me.
On the other hand it does not really surprise me, given all the mutating methods in MavenProject (set*) that seem like a bit of a mess in combination with concurrent access...

@kodeva Can you verify that this PR fixes your problem?

@famod - is this still needed please? The approach seems to work by @fkalinski...

famod · 2020-04-24T17:47:25Z

@famod - is this still needed please?

@kodeva No, thanks. I'll revisit this in the next days and I am going to try fkalinski@b38e3c2.
One difference in my setup that comes to my mind is that I don't have an aggregator plugin in the mix, AFAICS.

famod · 2020-04-25T20:07:06Z

Unfortunately, fkalinski@b38e3c2 does not fix my problem.
It seems there is yet another concurrency issue...

famod · 2020-04-26T20:27:12Z

I am now 100% sure that my problem is a different one. In my case resolvedDependencies is (sporadically) missing 4 transitive dependencies because resteasy-core is considered "invalid" due to missing dependency versions that come from a resteasy-BOM.
At the moment I am suspecting flatten-maven-plugin...

So, sorry for the confusion!

I'd just like to add that I found three other Jira tickets that seem to address the same error as MNG-6843 does:

famod · 2020-12-20T21:32:08Z

FYI @kodeva @fkalinski @rfscholte: I've created a new PR with my single ThreadLocal approach: #413

rfscholte · 2021-06-25T17:28:23Z

Superceded by #310

Clone projects in MavenSession.clone()

ce7be0e

Each build session will then have its own copy of MavenProject objects and cannot be influenced, for example, get determining compile classpath by other running build threads.

kodeva force-pushed the MNG-6843-missing-jars-in-parallel-build-classpath branch from abccf9a to ce7be0e Compare January 6, 2020 11:15

famod mentioned this pull request Apr 27, 2020

Sporadic dependency errors when building in parallel (mvn -T ...) mojohaus/flatten-maven-plugin#143

Closed

famod mentioned this pull request Dec 20, 2020

[MNG-6843] Thread-safe artifacts in MavenProject #413

Closed

lanmaoxinqing mentioned this pull request Jun 1, 2021

Parallel build fails due to missing JAR artifacts in compilePath apache/maven-mvnd#418

Closed

rfscholte closed this Jun 25, 2021

famod mentioned this pull request Dec 1, 2021

[MNG-7335] Parallel build fails due to missing JAR artifacts in compilePath #624

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MNG-6843] - Parallel build fails due to missing JAR artifacts in compilePath #310

[MNG-6843] - Parallel build fails due to missing JAR artifacts in compilePath #310

kodeva commented Jan 6, 2020 •

edited

Loading

rfscholte commented Jan 6, 2020

kodeva commented Jan 7, 2020

famod commented Apr 12, 2020 •

edited

Loading

rfscholte commented Apr 13, 2020

famod commented Apr 13, 2020 •

edited

Loading

famod commented Apr 13, 2020

famod commented Apr 13, 2020

famod commented Apr 13, 2020

fkalinski commented Apr 21, 2020

famod commented Apr 21, 2020

fkalinski commented Apr 21, 2020

fkalinski commented Apr 22, 2020

kodeva commented Apr 24, 2020

famod commented Apr 24, 2020 •

edited

Loading

famod commented Apr 25, 2020

famod commented Apr 26, 2020

famod commented Dec 20, 2020

rfscholte commented Jun 25, 2021

[MNG-6843] - Parallel build fails due to missing JAR artifacts in compilePath #310

[MNG-6843] - Parallel build fails due to missing JAR artifacts in compilePath #310

Conversation

kodeva commented Jan 6, 2020 • edited Loading

Description

rfscholte commented Jan 6, 2020

kodeva commented Jan 7, 2020

famod commented Apr 12, 2020 • edited Loading

rfscholte commented Apr 13, 2020

famod commented Apr 13, 2020 • edited Loading

famod commented Apr 13, 2020

famod commented Apr 13, 2020

famod commented Apr 13, 2020

fkalinski commented Apr 21, 2020

famod commented Apr 21, 2020

fkalinski commented Apr 21, 2020

fkalinski commented Apr 22, 2020

kodeva commented Apr 24, 2020

famod commented Apr 24, 2020 • edited Loading

famod commented Apr 25, 2020

famod commented Apr 26, 2020

famod commented Dec 20, 2020

rfscholte commented Jun 25, 2021

kodeva commented Jan 6, 2020 •

edited

Loading

famod commented Apr 12, 2020 •

edited

Loading

famod commented Apr 13, 2020 •

edited

Loading

famod commented Apr 24, 2020 •

edited

Loading