m2e plugin does not copy resources from src/main/resources, probably regression in 2022-12

[robserm]

I noticed that the newest Eclipse 2022-12 does not copy resources, this is a regression compared to Eclipse 2022-09 .
Steps to reproduce:
-import the attached project as "Existing Maven Project'
resources_problem.zip
-select file examples.stackmachine.RunnableStackMachine and from context menu choose "Run As-> Java Application"

the result is NPE:
Exception in thread "main" java.lang.NullPointerException: Cannot invoke "java.net.URL.toExternalForm()" because "location" is null
at java.desktop/javax.swing.ImageIcon.(ImageIcon.java:234)
at examples.stackmachine.RunnableStackMachine$ImageButton.(RunnableStackMachine.java:372)
at examples.stackmachine.RunnableStackMachine$PushButton.(RunnableStackMachine.java:300)
at examples.stackmachine.RunnableStackMachine.createStackButtonPanel(RunnableStackMachine.java:131)
at examples.stackmachine.RunnableStackMachine.createStackPanel(RunnableStackMachine.java:117)
at examples.stackmachine.RunnableStackMachine.(RunnableStackMachine.java:41)
at examples.stackmachine.RunnableStackMachine.main(RunnableStackMachine.java:392)

The problem is that the images from src/main/resources/examples/stackmachine/img were not copied to target/classes

The problem DOES NOT occur on 2022-09.

If you remove "Excluded" filter from project "Java Build Path" everything works fine, see the image below.

[glhez]

I do have the same issue: the rule includes all, but is excluded directly.

It seems the default value for exclusion is indeed "**" : https://github.com/eclipse-m2e/m2e-core/blame/master/org.eclipse.m2e.jdt/src/org/eclipse/m2e/jdt/internal/AbstractJavaProjectConfigurator.java#L551

Where it should be perhaps be "**/*.java".

Note sure while it fails now since the commit date from 6 years ago - probably the file were included in the runtime classpath of launch configuration ?

[HannesWell]

It seems the default value for exclusion is indeed "**" : https://github.com/eclipse-m2e/m2e-core/blame/master/org.eclipse.m2e.jdt/src/org/eclipse/m2e/jdt/internal/AbstractJavaProjectConfigurator.java#L551

I'm not entirely sure but I assume that the exclusions you are referring only affect the elements are (tried) to be compiled. And since resources are not compiled this makes sense.

The problem is that the images from src/main/resources/examples/stackmachine/img were not copied to target/classes

The problem DOES NOT occur on 2022-09.

Regarding copying resources, we had regressions in the M2E release contributed to 2022-12 SimRel:

• Resources handling is broken when there is a .mvn folder #1150
• [regression] MavenExecutionContext.execute(MavenProject, MojoExecution, IProgressMonitor) loses the MojoExecution configuration #1304

But they have been fixed in

• Fix wrong realm used when executing mojos #1161
• Properly clone mojoExecution #1305

Both available in the latest (just released) M2E release. Therefore can you please try the latest M2E release and check if the issue is already solved. The p2-repo is at:
https://download.eclipse.org/technology/m2e/releases/latest/

[robserm]

Confirmed, Eclipse 2022-12 works fine if m2e is upgraded to the latest version.
The issue can be closed. Thank you very much.

[HannesWell]

Great, thanks for checking it.

[glhez]

Hello @HannesWell

I tried the following version:

And then:

1. removed project from workspace
2. cleaned .project/classpath using git clean -fxd
3. reimported them

But the "**" is still there.

As it seems to work (resource was there after the clean), I'm curious of the change ?

Is this because potential maven plugins may perform copy (or filtering) of resources and the default copy of Eclipse can't work in those case ?

[laeubi]

@glhez I think you are misinterpreting what happening here.

lets say you have a source folder, what will happen:

1. JDT will put anything not excluded on the classpath, but never copies anything!
2. JDT will try to compile any java class and put it on the output folder (what might be different from what maven uses!)

So the exclude ** is used to make this folder visible to the user as a source folder, but disable the default JDT action

under the hood now, maven copies everything from the resource folder to the target/classes folder, and that content is also on the claspath, so now it "works", but the content is/was never used from the resource folder directly!

[glhez]

That's more or less what I understood. Thanks for the reply.

[martinlippert]

Based on some recent comments that we got at the Spring Tools issue tracker, it looks like the issue is still around, but occurs less often: spring-projects/sts4#929 (comment)

[laeubi]

@martinlippert if you can provide a test-case that reproduce the issue that would be good, otherwise its a bit hard to tell why it should still occur, especially if it does not happen always.

[glhez]

@laeubi , @martinlippert I remember running in debug Eclipse and stopping in debug near maven-resources-plugin (here: https://maven.apache.org/plugins/maven-resources-plugin/xref/index.html line 293)

My observation were that maven-resources-plugin perform the copy (files are found in explorer when browsing
target/classes) but something after delete the resources

If you use Alt + F5 (or "Maven configure"), the "Clean" checkbox is checked by default : try without it.
I also think that cleaning the whole workspace (as opposed to one single project, to force recompilation and copy of resources)

Observation aside, I don't know what is performing the clean (m2e, maven, Eclipse...).

[martinlippert]

@martinlippert if you can provide a test-case that reproduce the issue that would be good, otherwise its a bit hard to tell why it should still occur, especially if it does not happen always.

Agree, @BoykoAlex is also looking into this to see if we can provide more details here. The description in spring-projects/sts4#929 (comment) provides a few steps how to possibly reproduce the issue.

[BoykoAlex]

@laeubi there is a reproducible scenario described here: spring-projects/sts4#929 (comment)

I see that resources are not copied into the target folder as well as sometimes .class files are missing as well. If I debug JavaBuilder and MavenBuilder with its build participants (one of them is the mojo execution for the maven resources plugin) it all works fine. However, once breakpoints are skipped the issue occurs. Feels like a timing issue... i.e. downstream project is being built before upstream projects are finished building... perhaps a scheduling rule issue...

[laeubi]

That's quite vague, I think what would be required is a test-case that shows the problem in an isolated way so we can better investigate on it, also it might help to build the failing projects with -T64 to see if it is maybe a dependency issue but I must confess I can't see why a copy resources (what is local to the project) should interfere with other parts of the build here?

[BoykoAlex]

@laeubi There is a much simpler use case here: spring-projects/sts4#929 (comment) that reliably reproduces the issue where resources are missing from the target folder... In fact they are created but later removed (by the JavaBuilder#buildAll(...) i think)

Here is the JUnit you could paste into org.eclipse.m2e.core.ui.tests. Sorry for a few Thread.sleep(...). Had to put them in otherwise the test may fail at asserts after the pom file becomes invalid.

It is important to have Auto Build on!

Anyway, I hope this would provide you enough info to investigate matters on your end.

package org.eclipse.m2e.core.ui.tests;

import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;

import java.io.ByteArrayInputStream;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Path;

import org.apache.commons.io.FileUtils;
import org.eclipse.core.resources.IFile;
import org.eclipse.core.resources.IProject;
import org.eclipse.core.runtime.IPath;
import org.eclipse.jdt.core.IJavaProject;
import org.eclipse.jdt.core.JavaCore;
import org.eclipse.m2e.tests.common.AbstractMavenProjectTestCase;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;

public class AppPropsNotCopiedIntoTargetTest extends AbstractMavenProjectTestCase {
	
	private File projectDirectory;

	@Override
	@Before
	public void setUp() throws Exception {
		super.setUp();
	    setAutoBuilding(true);
		projectDirectory = new File(Files.createTempDirectory("m2e-tests").toFile(), "demo");
		projectDirectory.mkdirs();
		copyDir(new File("resources/projects/demo"), projectDirectory);
	}

	@Override
	@After
	public void tearDown() throws Exception {
		FileUtils.deleteDirectory(this.projectDirectory.getParentFile());
		super.tearDown();
	}

	@Test
	public void test() throws Exception {
		IProject project = importProject(new File(projectDirectory, "pom.xml").toString());
		waitForJobsToComplete(monitor);
		Thread.sleep(5000);

		IJavaProject jp = JavaCore.create(project);

		assertTrue(isAutoBuilding());

		IPath rootPath = project.getWorkspace().getRoot().getLocation();
		
		// Project imported for the first time and built - everything is properly in the target folder
		assertTrue(rootPath.append(jp.getOutputLocation()).append("/application.properties").toFile()
				.exists());
		assertTrue(rootPath.append(jp.getOutputLocation())
				.append("/com/example/demo/DemoApplication.class").makeAbsolute().toFile().exists());

		IFile pomXml = project.getFile("pom.xml");
		String content = Files.readString(Path.of(pomXml.getLocationURI()));
		pomXml.setContents(new ByteArrayInputStream("Nothing".getBytes()), true, false, null);

		waitForJobsToComplete(monitor);
		Thread.sleep(5000);
		// Invalid pom file. JavaBuilder built java sources but MavenBuilder wqs unable to do anything hence no resources in the target folder except for compiled classes
		assertFalse(rootPath.append(jp.getOutputLocation()).append("/application.properties").toFile()
				.exists());
		assertTrue(rootPath.append(jp.getOutputLocation())
				.append("/com/example/demo/DemoApplication.class").toFile().exists());

		pomXml.setContents(new ByteArrayInputStream(content.getBytes()), true, false, null);
		waitForJobsToComplete(monitor);
		Thread.sleep(5000);
		// Valid pom file. Compiled classes and resources should reappear in the target folder. However, resources are missing which makes the test fail
		assertTrue(rootPath.append(jp.getOutputLocation())
				.append("/com/example/demo/DemoApplication.class").toFile().exists());
		assertTrue(rootPath.append(jp.getOutputLocation()).append("/application.properties").toFile()
				.exists());		
	}

}

[laeubi]

@BoykoAlex can you provide the test as a PR so we can maybe add it to the code? The sleep should maybe something like to wait for all autobuilds to complete @mickaelistria @HannesWell do we already have a helper method for that?

[BoykoAlex]

@laeubi I have created the PR with the unit test #1478

[HannesWell]

@BoykoAlex can you provide the test as a PR so we can maybe add it to the code? The sleep should maybe something like to wait for all autobuilds to complete @mickaelistria @HannesWell do we already have a helper method for that?

Yes we do, but it looks like @BoykoAlex already found AbstractMavenProjectTestCase.waitForJobsToComplete() by himself. :)

[pcdavid]

It seems this appeared in 2022-12 (we're seeing this in versions or Spring Tool Suites corresponding to that, but other colleagues who stayed on earlier versions of Eclipse/STS never see the bug).

2022-12 corresponds to Eclipse Platform 4.26, could this be related to the JobManager implementation changes in that version which maybe modified the way build jobs are scheduled?

[pcdavid]

I'm not familiar enough with m2e, but maybe @BoykoAlex's unit test could be tested with a target platform using Eclipse 4.25 and 4.26 to confirm this?

[kysmith-csg]

I'm actually getting a similar issue in 2023-03 and beyond, probably as a result of the lifecycle execution changes that were made. I've had to hold us back to 2022-12 because of this.

We are seeing the same thing, that things are sometimes copied but at some point deleted, then the launch fails because things are missing. Unfortunately we use a custom builder as well so I don't know how to isolate it, but it has always worked before (and continues to work with 2022-12).

We have custom file types that exist in other src/main folders, as well as things in src/main/resources. Java files in src/main/java are correctly compiled and placed into target/classes as before, but our other things (including resources) do not.

With autobuild off, selecting "Build Project" causes those files to be deleted.

@laeubi Please let me know if this seems related, otherwise I can open a separate issue for it.

[laeubi]

@kysmith-csg

Please try out the latest snapshot build, you can use this p2 repository:

https://download.eclipse.org/technology/m2e/snapshots/latest/

this should already improve the situation here.

probably as a result of the lifecycle execution changes that were made. I've had to hold us back to 2022-12 because of this.

This has nothing to do with lifecycle execution, all mojos are executed, but is an intrinsic problem of how m2e handles incremental builds that now becomes prominent as we are skipping incremental builds for changes in target folders (what previously in some cases lead to infinite build loops), made even worse by the way how the resources plugin works see:

• [MSHARED-1285] use an up-to-date scanner instead the newscanner apache/maven-filtering#77

so if anyone wants to help resolving that issue that should improve the situation already for older m2e versions.

[kysmith-csg]

I am using the latest snapshot for this purpose and still seeing issues. From my console, I can see:

1. A clean build is scheduled when I do Project>Clean. Files in target/ are removed and a Maven build is started, as well as our own custom builder.
2. At some point, ANOTHER build is scheduled. This one is an auto build (= clean build), so files in target/ are removed yet again. Except this time, our builder thinks that nothing is changed so things are not rebuilt.

I think part of the issue might be bad timing of our build. Basically,

1. Things in src/main/eventflow (our own source directory) and src/main/resources are copied to target/classes.
2. Our builder generates Java source files based on the files in src/main/eventflow and places them in target/generated-sources.

I think 2 ^^ above is triggering the autobuild because they are source folders being added. These are supposed to be generated by our builder and then built by JDT. But in this case, our builder is creating them, then something sees new source folders, and triggers another build (thus deleting everything that was made previously), but our builder doesn't think anything is different.

The source generation is normally called by the mojo generate-main-eventflow in TIBCOSoftware/tibco-streaming-maven-plugin, but this execution is skipped in the IDE because we provide a builder that does it directly.

The goal compile-eventflow-fragment is what copies the files over from src/main/eventflow to target/classes, and this one IS run by m2e.

What I think SHOULD be happening is,

1. m2e runs compile-eventflow-fragment and other mojos to copy resources.
2. Our builder generates Java sources and places them in target/generated-sources.
3. These generated sources are built by JDT WITHOUT triggering a clean build.

Does this make sense? Sorry for the long comment :) Is there anything that we need to be doing differently to make this work? Again, this used to work with m2e 2.1.0 (which is why I thought it was the mojo execution changes in 2.2.0).

[laeubi]

but our builder doesn't think anything is different.

Lets say you has input file In and your builder produces an output file Out it should always use BuildContext#isUptoDate(In, Out), then it will notice that the output is gone. JDT itself should never delete the target folder but target/classes as this is the binary output folder, if it happens you most probably have configured your project wrong!

[kysmith-csg]

Yes, sorry if I misspoke, the target folder itself is never deleted (nor target/classes actually). Just the contents. But the confusing part is why these files are being deleted after the build.

When I do perform clean build: JDT builds, then m2e, then our builder, and repeat. I think it runs twice because our builder is generating Java files, thus modifying the classpath.

The 2nd time JDT runs, all the output files get deleted (because it is running auto build), but the rest of the build (for example, resource copying) is not run again by m2e. Using m2e 2.1.0, JDT does not delete these output files the 2nd time. So it appears as if the files are not being copied over, but in reality they are, but just deleted right after.

[laeubi]

The 2nd time JDT runs, all the output files get deleted (because it is running auto build), but the rest of the build (for example, resource copying) is not run again by m2e.

This is possible because from m2e POV nothing has changed, the problem is that JDT deletes all content not only class files...

Using m2e 2.1.0, JDT does not delete these output files the 2nd time.

Have you really checked this? In older versions the deletion and creation of classfiles has just scheduled another build (what in the worst case scheduled another and so on ...)

[kysmith-csg]

Pretty sure yeah, although there's a lot going on so might be missing something. I have a resource listener which prints deltas to the console. With m2e 2.1, these files report a REPLACED delta (meaning they are still there after), and a REMOVED delta in 2.2+. The reason I say that JDT is the one deleting them is because I put a breakpoint in IResource.delete that breaks for one of the files I expect to be there, and the stack leads back to JavaBuilder.clean (coming from an auto build). I'll see if it's possible to make a reproducer without having to have our ide. This would help me know too whether we are doing something wrong regarding builds.

…

On Monday, August 7, 2023, Christoph Läubrich ***@***.***> wrote: The 2nd time JDT runs, all the output files get deleted (because it is running auto build), but the rest of the build (for example, resource copying) is not run again by m2e. This is possible because from m2e POV nothing has changed, the problem is that JDT deletes *all* content not only class files... Using m2e 2.1.0, JDT does not delete these output files the 2nd time. Have you really checked this? In older versions the deletion and creation of classfiles has just scheduled another build (what in the worst case scheduled another and so on ...) — Reply to this email directly, view it on GitHub <#1302 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BARNQSXHML3JDTOO64TABDTXUGUUVANCNFSM6AAAAAAVQ6CFPA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Kyle Smith | *M* +* *** *** **** | *E* ***@***.*** | www.tibco.com <http://www.tibco.com/>

[tarioch]

Can this issue be reopened? Would make it a bit easier to find it and figure out that this is not yet fixed.

[laeubi]

Most of the recent discussion has nothing to do with the initially reported problem, so it would be better to open new dedicated issue for what could be seen with using the latest snapshot build:

https://download.eclipse.org/technology/m2e/snapshots/latest/

[tarioch]

Created #1511

[kysmith-csg]

For a bit of an update, I debugged various builders and things for almost an entire day. What was happening was that our builder, on clean(), was deleting a generated-sources/src that it put in target previously. This generated-sources/src is the output of our builder but it is also a source folder because we want JDT to compile these generated Java files.

The deletion of this folder triggers an incremental build, but JDT changes their own builder to full since it sees a classpath change (JavaBuilder.hasClasspathChanged()). Our builder of course was none the wiser and still tried to run an incremental build, doing nothing because it didn't know JDT did this.

The fix was to delete the contents of generated-sources/src without deleting the folder itself, that way the classpath didn't change and JDT continued to keep using an incremental build here.

What remains unclear is why updating to m2e 2.2.0 caused issues where it had been working fine like this forever.