Loop between "Building" and "Downloading Sources and Javadoc"

[mickaelistria]

While I'm coding for Tycho, on snapshot branch, I see a lot of CPU used by Eclipse IDE and the Progress view seems to loop between "Building" and "Downloading Sources and Javadoc" pretty often.

[mickaelistria]

It might just be a symptom of #123

[laeubi]

Yep that seems to be a reincarnation of that bug. As mentioned there, it some time magically 'happens' but goes away without a clear indication why.

[mickaelistria]

That typically happens when having some Tycho modules (incl tycho-core and most of content of tycho-bundles) open, but not all, and opening a new module.

[reckart]

Happens for me a lot as well after updating to m2e 1.18.1 - different workspaces, some with Maven modules that are bundles, some only containing regular JAR/WAR/POM Maven modules.

[reckart]

Downgrading to m2e 0.17.2 seems to fix the issue.

[mickaelistria]

I merged a patch that may improve things yesterday. Maybe it's also worth trying the latest snapshot of 1.18.2.

[reckart]

Do you have the update site URL handy?

[mickaelistria]

See https://github.com/eclipse-m2e/m2e-core/blob/master/CONTRIBUTING.md#-trying-latest-builds

[reckart]

I have installed m2e - Maven Integration for Eclipse (includes Incubating components) 1.18.2.20210721-064 - loops are still there. The funny thing is that "Download Artifact Sources" and "Download Artifact JavaDoc" are even both turned off in the preferences. "automatically update configuration" is also off. Still it loops on "Download sources and javadoc" <-> "Building" until I manually break the loop by killing the "Download sources and javadoc" task.

[mickaelistria]

@reckart thanks for trying. Do you have a minimal project that can reproduce this issue in a predictable manner?

[reckart]

Unfortunately not.

[a1dutch]

m2e - Maven Integration for Eclipse (includes Incubating components) 1.18.2.20210721-0646 fixed the endless building/downloading sources for my maven projects

[ciros88]

I confirm build 1.18.2.20210721-0646 fixed that annoying problem

[HannesWell]

I think the actual cause of this problem is that after the sources/javadoc of a project were downloaded that classpath is updated and during the classpath update a download of the attached sources/javadoc is triggered again.
The following stack-trace illustrates this (a BuildPathManager references its DownloadSourcesJob and vice-versa).

	DownloadSourcesJob.addDownloadRequest(IProject, IPackageFragmentRoot, ArtifactKey, boolean, boolean) line: 352	
	DownloadSourcesJob.scheduleDownload(IProject, IPackageFragmentRoot, ArtifactKey, boolean, boolean) line: 338	
	DownloadSourcesJob.scheduleDownload(IProject, ArtifactKey, boolean, boolean) line: 361	
	BuildPathManager.scheduleDownload(IProject, MavenProject, ArtifactKey, boolean, boolean) line: 899	
	BuildPathManager.configureAttachedSourcesAndJavadoc(IMavenProjectFacade, Properties, ClasspathDescriptor, IProgressMonitor) line: 342	
	BuildPathManager.getClasspath(IMavenProjectFacade, int, Properties, boolean, IProgressMonitor) line: 269	
	BuildPathManager.getClasspath(IProject, int, boolean, IProgressMonitor) line: 393	
	BuildPathManager.getClasspath(IProject, int, IProgressMonitor) line: 373	
	BuildPathManager.getClasspath(IProject, IProgressMonitor) line: 401	
	BuildPathManager.updateClasspath(IProject, IProgressMonitor) line: 205	
	DownloadSourcesJob.updateClasspath(BuildPathManager, Set<IProject>, Map<IPackageFragmentRoot,Attachments>, IProgressMonitor) line: 209	
	DownloadSourcesJob.run(IProgressMonitor) line: 187	
	Worker.run() line: 63

While Mickaels change fixed the problem for the most cases the logic in BuildPathManager.configureAttachedSourcesAndJavadoc leads to an endless loop in case for a SNAPSHOT version of a maven artifact the sources were last-modified before the main-artifact (don't ask me how this happened in my local Maven repo, but it happened).
So in my case the DownloadSources-job runs forever every second and sleeps in between. But I think this can be fixed easily if a job maintains a Set of DownloadRequest of the current run and checks if a request about to be added to the queue is in that set.
This way we should be safe in all circumstances.

I'm about to prepare a PR for this.

[rgambelli]

Happens for me a lot as well after updating to m2e 1.18.1 - different workspaces, some with Maven modules that are bundles, some only containing regular JAR/WAR/POM Maven modules.

It is exactly what happening to me too, yesterday I've done a full upgrade in my Eclipse STS and m2e now is:
Eclipse.org - m2e Maven Integration for Eclipse 1.18.1.20210618-2246 org.eclipse.m2e.feature

and the issue appers systematically

[mickaelistria]

What would really help more than "me too" would be if someone can share a reproducible example that allows to reproduce the issue deterministically.

[HannesWell]

It is exactly what happening to me too, yesterday I've done a full upgrade in my Eclipse STS and m2e now is:
Eclipse.org - m2e Maven Integration for Eclipse 1.18.1.20210618-2246 org.eclipse.m2e.feature

and the issue appers systematically

Your version is the latest release. Have you tried to update to the latest snapshot? It can be found in this p2-repo:
https://download.eclipse.org/technology/m2e/snapshots/1.18.2/latest/

With the latest snapshot the loop only occurs in the rare situation I have described above, which probably happened to me because I was installing artefacts into my own Maven repo.

[rgambelli]

Really thanks @HannesWell, today I've upgraded to m2e 1.18.2.20210826-0749 and that fixed the issue 👍

[HannesWell]

I think with Mickaels change (I think it was 9c0c5f8) and with the merge of PR #303 this issue is resolved and can be closed.

[adam-law]

I just installed Eclipse 09-2021 and this is still happening though.

[laeubi]

I think this is because m2e do not contributed a new release @mickaelistria ?

[mickaelistria]

Right, m2e was not released since this was fixed; I need to investigate about doing a new release shortly, although IIRC it requires first a new release of Wild Web Developer.

[HannesWell]

Regarding a release of m2e I want to mention that the pde integration now automatically converts Maven-targets in a target-platform file into the new format that is more closely to the Maven dependency definition and also supports more elements like repositories. But those Maven-targets are not supported before Tycho 2.5.0 which is currently under development.

Therefore I suggest to not release m2e before Tycho 2.5.0 has been released. Otherwise m2e would create target files Tycho cannot handle or users are forced to use Tycho snapshot when they use the latest m2e release.

[famod]

I'm still having this problem with Quarkus (even with 1.18.2, fresh Eclipse 2021-09):

• clone https://github.com/quarkusio/quarkus
• ./mvnw -Dquickly -T0.5C (will take a while, > 900 modules!)
• import e.g. core/deployment and test-framework/junit5
• loop (sooner or later)

[mishka81]

Where do I get version 1.18.2 or 1.18.3 ?
URLs
https://download.eclipse.org/technology/m2e/snapshots/1.18.X/latest/

Are not working ...

[HannesWell]

You can get them from here:

• https://download.eclipse.org/technology/m2e/releases/1.18.2/
• https://download.eclipse.org/technology/m2e/releases/1.18.3/

Those older releases are archived, but still available under the original address. If you open the URLs in the browser you will be re-directed to the archive were you find all releases in the archive.
https://download.eclipse.org/technology/m2e/releases contains the most recent releases but from time to time older ones are moved to the archive.

[mishka81]

If I open the URL in browser I have this :

In Eclipse when adding the update site I have this :

[HannesWell]

Indeed the 1.18.3 does not exist (and then never did). But the 1.18.2 does. If you click on that link you will see the redirect and P2 can work with that directly.
Here is an overview of all archived releases:
https://archive.eclipse.org/technology/m2e/releases/

[RichMacDonald]

Using M2E version 2.3.0:

I have a jar built with the "maven-install-plugin". It includes both the class jar and a sources jar. Except that the sources jar is badly built (the source is inside a "Classes" folder" inside of the root). When m2e tries to resolve this in the local repository, it leads to an infinite loop.

[mickaelistria]

@RichMacDonald Thanks. Can you please try to build a minimal project that can reproduce this issue and open a new ticket with the steps to reproduce?

[RichMacDonald]

@mickaelistria Reproducing is problematic: I made a standalone maven project using just the offending jars. Of course, it worked fine. Grrr.

This doesn't surprise me. The problem has always been intermittent.

1. Worse on larger projects.
2. Often triggered by a modification of the project structure, such as adding a new module that sends it temporarily into a bad state (the new module is briefly configured as a maven project but not added to the modules list, or vice-versa).
3. Eclipse->Project->Build-Automatically at the same time as running a maven command.

I have tried turning on all the maven configuration settings to refresh everything on startup. I have tried turning them off. The problem occurs in both.

If I get lucky on something, I will report back. Otherwise, I am very sorry. No can reproduce.

[reckart]

Is there any option that can be enabled to log reasons for triggering a rebuild?

[HannesWell]

Is there any option that can be enabled to log reasons for triggering a rebuild?

You can use the Maven Workspace Build view in the IDE to show the resource-deltas that triggered a build.
Additionally opening a Maven Console in the Console View maybe could also be helpful in this regard.

[reckart]

@HannesWell that view sounds helpful. I found it, opened it, and triggered an incremental build by a small change in a file as well as a rebuild using "Maven -> Update project" - but both did not register in the view. Do I need to do anything extra to make the view register builds?

[laeubi]

The view has a "paused" (yellow) icon you need to first enable recordings there.

[reckart]

@laeubi Ah, thanks! I had actually noticed the pause button, but I did not think that I had to press it. Might be worth considering to open the view in an unpaused mode as to avoid users being confused about the view not doing anything?

[AndreasFagschlunger]

I run into this issue a lot during my work. It mostly happens when I'm working on projects of one big maven project with around 300 modules. I only import the few projects I need and sometimes this whole loop starts. The Maven Workspace Build view only showed one projects which was rebuilding over and over. Is there a way I can provide information for solving this issue?

[laeubi]

Is there a way I can provide information for solving this issue?

See

• What is your favorite maven project that is NOT working with m2eclipse? #1064

for a list of useful information in such case, "one big maven project with around 300 modules" is very hard to investigate on :-)

Of course you should make sure to use the latest m2e release and maybe can extract some information from the mentioned view what is triggering the rebuild.

[reckart]

I have one of these large projects (actual several) and while I do not have the infinite build problem right now (I have it sometimes), I can see that the parent module (type pom) of which all other modules are submodules has a very high build count. Opening it, I see that its delta section includes basically all the files from the target folders of submodules. Does that make sense? I mean, if a file is already triggering a delta build of a submodule, should it also trigger a delta build of higher-level reactor modules?

[laeubi]

It depends on the delta, basically m2e assumes that if the parent changes there is a high chance that childs are affected (otherwise one won't reference it as a parent), since the last release m2e has some improvements to not build unnecessarily, but this might need still improvements.

[reckart]

You say if the parent changes, the child needs to be build - that sounds correct.

But if I read the view correctly, then the parent is built because files in the target folders of its submodules change - that seems odd. Maybe I am readying the view wrong - I assume the files listed under the delta section are the triggers for the build and not the files updated by the build.

[laeubi]

Yes these are triggers for a build, but due to the eclipse resource-model m2e will receive deltas for all child items of the parent project so you will see a long list here, but this should not trigger the build as m2e ignores such deltas here if the occur in a submodule:

m2e-core/org.eclipse.m2e.core/src/org/eclipse/m2e/core/internal/builder/MavenBuilderImpl.java

Line 185 in 8c838c8

private boolean hasRelevantDelta(IMavenProjectFacade projectFacade, IResourceDelta resourceDelta)

if you can't share your project here, the best bet is to start Eclipse with remote debugging enabled (via eclipse.ini) and then attach a debugger from another instance (you don't need a full m2e workspace for that, just the sources of interest and even those not need to compile for debugging). As said the feature is quite new and there might be cases where it still fails to detect, also the test coverage is very low in the area of "rebuilds", so adding test-cases for an identified issue would be great as well.

[reckart]

Sorry for using this as a support/feature request thread... but is there some way of seeing through which way a file was scheduled as a delta trigger? For example, I have some files here in src/main/resources appearing in the delta section, But these particular files should be static (i.e. they should not be touched/written to in any way during builds). Yet they have an increased count in the build # column - which I interpret as them having triggered as many incremental builds - which seems very strange if they are static.

[laeubi]

but is there some way of seeing through which way a file was scheduled as a delta trigger

Not really, depending on your OS you might want to use native tools to watch this particular file (e.g. inotify https://serverfault.com/a/780522 ), basically if the file is touched (even if content stays the same) it is assumed as a delta.

[reckart]

As said the feature is quite new and there might be cases where it still fails to detect, also the test coverage is very low in the area of "rebuilds", so adding test-cases for an identified issue would be great as well.

I'll update my m2e to the lastest version before making any further observations... what I am seeing now is old behavior.

[laeubi]

By the way, there is one bug in resource plugin itself that make things a bit more worse but it has not gotten much attention yet:

• [MSHARED-1285] use an up-to-date scanner instead the newscanner apache/maven-filtering#77

[reckart]

After upgrading to m2e 2.3.0, numbers in the build view look considerably more sane. Although in many cases, I now see entries that have neither delta nor execution children - so it is unclear what triggered those builds.

[reckart]

... it also seems that many builds are not recorded anymore at all ... E.g. I see builds happing via the progress view, but they do not seem to be added to the Maven Workspace Build view.

[laeubi]

"build" is not a exclusive m2e term, it could be anything, including java source compiles, verification and so on...

[reckart]

Ok, but I clearly can see "Invoking Maven Project Builder" flashing through rapidly in the progress view but the "Maven Workspace Build" view does not seem to be updated accordingly.

[AndreasFagschlunger]

For me the "Maven Workspace Build" doesn't show anything. The loop in the "Progress" view is:

(m2) Download sources and javadoc...
Associate javadoc and sources with classpath - <project 1-n>

Then a few

JPA Project Change Event Handler (Waiting) ...

then

Building <project x>

And then it starts over with downloading Javadoc & Sources.
It loops pretty fast, is there a log of the progress available?

If I disable "Build automatically" the "Building" step is gone, but it loops between "Download sources and javadoc" and "JPA Project Change Event Handler".