Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Classpath scanning does not work in tests using the module path [SPR-16977] #21515

Closed
spring-projects-issues opened this issue Jun 26, 2018 · 26 comments
Assignees
Labels
in: core Issues in core modules (aop, beans, core, context, expression) in: test Issues in the test module status: superseded An issue that has been superseded by another

Comments

@spring-projects-issues
Copy link
Collaborator

spring-projects-issues commented Jun 26, 2018

Andy Wilkinson opened SPR-16977 and commented

It appears the classpath scanning doesn't work when Surefire launches the JVM configured to use the module path. target/classes is placed on the module path and target/test-classes is patched into this module but classpath scanning only finds classes in target/test-classes.

I have attached a minimal sample that should reproduce the problem when built (mvn test) with Java 10. The sysout from the test should show that only the test class has been found:

[INFO] Running com.example.ScanningTest
[file [/Users/awilkinson/dev/temp/module-path-scanning/target/test-classes/com/example/ScanningTest.class]]

To my rather untrained eye, building with -X and examining the arguments that Surefire uses to launch the forked JVM (in target/surefire) suggests that Surefire's configuration of the JVM is correct.

When the sample is modified to work with Java 8 (remove module-info.java and change the compiler plugin configuration to remove <release>10</release>) the class in target/classes is also found:

[INFO] Running com.example.ScanningTest
[file [/Users/awilkinson/dev/temp/module-path-scanning/target/test-classes/com/example/ScanningTest.class], file [/Users/awilkinson/dev/temp/module-path-scanning/target/classes/com/example/One.class]]

Affects: 5.0.7

Reference URL: spring-projects/spring-boot#13581

Attachments:

Issue Links:

2 votes, 7 watchers

@spring-projects-issues
Copy link
Collaborator Author

spring-projects-issues commented Jun 26, 2018

Juergen Hoeller commented

This all depends on ClassLoader.getResources results for the given base package. The module system possibly only exposes the root URL for the patched part there? In any case, debugging ClassLoader.getResources results in both scenarios would help here...

@spring-projects-issues
Copy link
Collaborator Author

spring-projects-issues commented Jun 27, 2018

Andy Wilkinson commented

On closer inspection, this appears to be a bug in the JDK. With the test changed to the following:

@Test
public void scanningTest() throws Exception {
    Enumeration<URL> resourceUrls = getClass().getClassLoader().getResources("com/example");
    while (resourceUrls.hasMoreElements()) {
        System.out.println(resourceUrls.nextElement());
    }
    PathMatchingResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
        System.out.println(Arrays.toString(resolver.getResources("classpath*:com/example/**/*.class")));
}

It produces output similar to this:

[INFO] Running com.example.ScanningTest
file:/Users/awilkinson/dev/temp/module-path-scanning/target/test-classes/com/example/
file:/Users/awilkinson/dev/temp/module-path-scanning/target/test-classes/com/example
[file [/Users/awilkinson/dev/temp/module-path-scanning/target/test-classes/com/example/ScanningTest.class]]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.094 s - in com.example.ScanningTest

Note the two (slightly different) URLs that are both pointing to target/test-classes. This explains why no classes in target/classes are found.

@spring-projects-issues
Copy link
Collaborator Author

spring-projects-issues commented Jun 27, 2018

Juergen Hoeller commented

Looks like a bug in the ClassLoader implementation indeed. They might not have fully tested this in module patch scenarios...

Probably worth reporting to OpenJDK. Aside from a proper fix, maybe there's a workaround to discover in time for our GA still.

@spring-projects-issues
Copy link
Collaborator Author

spring-projects-issues commented Jun 27, 2018

Andy Wilkinson commented

I've reported a bug. It's in internal review at the moment (ID 9055803). I'll comment again here if it makes it out the other end and becomes public. For reference, this is the test case that I provided with the bug report. Unlike the sample attached to this issue, it does not use Spring Framework.

@spring-projects-issues
Copy link
Collaborator Author

spring-projects-issues commented Jul 3, 2018

Andy Wilkinson commented

The bug is now public but has been closed as not an issue. The situation's now being discussed in this thread on the core libs dev OpenJDK mailing list.

@spring-projects-issues
Copy link
Collaborator Author

spring-projects-issues commented Jul 3, 2018

Andy Wilkinson commented

The thread on the mailing list seems to have reached a conclusion. The following recommendation came from Alan Bateman:

With modules then it should be looking at the modules in the boot layer and using ModuleReader to get the contents. It can use the value of java.class.path to scan the class path.

@spring-projects-issues
Copy link
Collaborator Author

spring-projects-issues commented Dec 16, 2018

Thomas Kratz commented

Are there any plans to resolve this like recommended? 

Would doFindAllClassPathResources(String path) be the point to put that? I don't clearly understand that suggestion, but maybe I could invest some time over the holidays.

I would still think this is a JDK issue.

 

@wimdeblauwe
Copy link

wimdeblauwe commented Sep 9, 2019

Is there a workaround for this? I have a JavaFx project that I added Spring Boot to which works really great, but I am unable to run my @JsonTest.

As a workaround, I need to manually specify the application class + the json deserializers I want to test:

@JsonTest
class MyJsonDeserializerTest {
   ...
}

needs to become:

@JsonTest
@ContextConfiguration(classes = {MyApplication.class, MyJsonDeserializerTest.TestConfig.class})
class MyJsonDeserializerTest {
   ...

    @TestConfiguration
    static class TestConfig {
        @Bean
        public MyJsonDeserializer deserializer() {
            return new MyJsonDeserializer();
        }
    }

}

(Note: It seems I only need to do this for Maven, IntelliJ seems to run the test fine, but I have to assume that IntelliJ is somehow "cheating" given this issue and the related spring-projects/spring-boot#13581 )

@eiswind
Copy link

eiswind commented Dec 28, 2019

Another year has passed.
To me this still is a showstopper.

@tgolden-andplus
Copy link

tgolden-andplus commented Feb 19, 2020

I have the opposite problem currently. Failsafe works just fine, apparently because they point to the packaged JAR instead of the exploded target/classes directory. However, IntelliJ wants to use the exploded directory when running integration tests in the IDE, so this fails unless I uncheck the module-path option (which I have to do for every run configuration separately).

Additionally, I can make Failsafe break the same way as the IDE by forcing it to use the exploded directory (via <classesDirectory>${project.build.outputDirectory}</classesDirectory>), so there's something magic about being in a JAR versus hanging around on the filesystem -- but I'm not clear on what.

Is this truly a JDK issue or a Spring issue? It doesn't seem that the core JDK team plans on changing behavior, thus it seems incumbent on the Spring team to work around this somehow (or at the very least, document that component scanning will break unexpectedly when using exploded classes versus packaged in JAR classes....)

@tomdw
Copy link

tomdw commented Feb 25, 2020

@jhoeller @wilkinsona could this be fixed inside spring by making the component scanning code be more module path aware? Especially for running spring tests on module path using the surefire plugin or in intellij this is a problem.

@petermylemans
Copy link

petermylemans commented Mar 7, 2020

@tgolden-andplus I'm having the same issue.

For future reference; the only work around I found was to uncheck "use module path" in Templates > JUnit in your Run/Debug Generations to make sure it is unchecked for every new junit run config you create.

@manosbatsis
Copy link

manosbatsis commented Mar 11, 2020

For me the workaround was using a org.springframework.core.io.ClassPathResource instead of ClassLoader.getResource

@tomdw
Copy link

tomdw commented Mar 11, 2020

@manosbatsis and how dit you make sure that spring uses this ClassPathResource for its component scanning?

@tgolden-andplus problem with that workaround is that you then no longer have the module checking which is important. Also stops working when using e.g. serviceloader to load services from other modules in your code under test

Probably the only fix is a fix in the code of spring itself, solving the way it does component scanning to handle modules better.

@manosbatsis
Copy link

manosbatsis commented Mar 11, 2020

@tomdw to be clear, i had trouble when trying to "manually" load a classpath resource from within tests using ClassLoader.getResource. After a search landed me here, i figured out a workaround was needed to bypass the bug and ClassPathResource did the trick for me, so i thought it might be useful.

@retheesh-mr
Copy link

retheesh-mr commented Apr 14, 2020

I am facing similar issue when writing junit5 cases with spring boot and java 11. As the jdk bug raised is in resolved status, what is further action on this? How can this be resolved?
It seems module-info cannot be used together with junit and spring due to this issue. As it is blocking for me in my project I have no other way than to fall back to java 8 version.

@kopper
Copy link

kopper commented Jun 30, 2020

I had the same (or very similar) issue running tests with maven-failsafe-plugin 3.0.0-M5 on AdoptOpen JDK 11. The work around is to disable using module path in plugin's configuration
<useModulePath>false</useModulePath>

@oscarhaglund
Copy link

oscarhaglund commented Oct 5, 2020

I should mention that I am also suffering from this issue and it would really be appreciated if this could be solved by Spring if it is not going to be solved by Java itself.

@prakash0409
Copy link

prakash0409 commented Jan 12, 2021

We upgraded to a new version of spring boot 2.3.4 and all our integration tests started failing. All our integration test cases were earlier working on AdoptOpen JDK 11 with the older version. I guess this should be an issue at Spring end. As mentioned above the classpath scanning is only finding only "target/test-classes".

I was able to use the workaround as suggested by @kopper

@msche
Copy link

msche commented Jan 30, 2021

FYI, I had similar classpath issues when I introduced java modules in my project and executed the Spring Boot unit tests.

07:28:35.820 [main] ERROR - o.s.test.context.TestContextManager - prepareTestInstance - Caught exception while allowing TestExecutionListener [org.springframework.test.context.support.DependencyInjectionTestExecutionListener@63e5b8aa] to prepare test instance [...]
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name '...': Unsatisfied dependency expressed through field '...'; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type '...' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}

I eventually got the test executed successfully by including within the configuration of SureFire version 2.22.x <forkCount>0</forkCount>.

@barbarosalp
Copy link

barbarosalp commented Apr 5, 2021

Is there any proper solution, we are also stuck with this?

@OleksandrGavryliukTR
Copy link

OleksandrGavryliukTR commented Apr 23, 2021

FYI, I had similar classpath issues when I introduced java modules in my project and executed the Spring Boot unit tests.

07:28:35.820 [main] ERROR - o.s.test.context.TestContextManager - prepareTestInstance - Caught exception while allowing TestExecutionListener [org.springframework.test.context.support.DependencyInjectionTestExecutionListener@63e5b8aa] to prepare test instance [...]
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name '...': Unsatisfied dependency expressed through field '...'; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type '...' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {@org.springframework.beans.factory.annotation.Autowired(required=true)}

I eventually got the test executed successfully by including within the configuration of SureFire version 2.22.x <forkCount>0</forkCount>.

This has fixed it for me. With a warning:
[WARNING] useSystemClassloader setting has no effect when not forking
Which must be removing the cause of this problem.

@jhoeller jhoeller self-assigned this Nov 1, 2021
@jhoeller jhoeller modified the milestones: 6.x Backlog, 6.0 M1 Nov 1, 2021
@jhoeller jhoeller modified the milestones: 6.0 M1, 6.0 M2 Dec 7, 2021
@jhoeller jhoeller modified the milestones: 6.0.0-M2, 6.0.0-M3 Jan 5, 2022
@jhoeller jhoeller removed their assignment Jan 24, 2022
@sbrannen sbrannen modified the milestones: 6.0.0-M3, 6.0.0-M4 Mar 15, 2022
@sbrannen sbrannen self-assigned this Mar 15, 2022
@sbrannen sbrannen added the in: test Issues in the test module label Mar 15, 2022
@sbrannen sbrannen modified the milestones: 6.0.0-M4, 6.0.0-M5 May 9, 2022
@sbrannen
Copy link
Member

sbrannen commented May 16, 2022

Disclaimer: This is in no way production ready code. This is 100% "proof of concept" code.

Now, having said that... with an updated version of the original example application from @wilkinsona (now using Java 17, Maven 3.8.5, Maven Surefire 3.0.0-M6, Spring Framework 5.3.20, and JUnit Jupiter 5.8.2), I have come up with the following.

class ScanningTest {

	@Test
	void scanningTest() throws Exception {
		PathMatchingResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
		System.err.println(Arrays.toString(resolver.getResources("classpath*:com/example/**/*.class")));

		System.err.println("------------------------------------------------");

		String moduleName = getClass().getModule().getName();
		List<String> resourceNames = streamResolvedModules(Predicate.isEqual(moduleName))//
				.map(ResolvedModule::reference)//
				.map(moduleReference -> scanForNames(moduleReference, "^com/example/.+\\.class$"))//
				.flatMap(List::stream)//
				.toList();

		resourceNames.forEach(System.err::println);

		System.err.println("------------------------------------------------");

		resourceNames.forEach(resouceName -> {
			String className = ClassUtils.convertResourcePathToClassName(resouceName);
			className = className.substring(0, className.length() - ".class".length());
			try {
				Class<?> clazz = ClassUtils.forName(className, getClass().getClassLoader());
				System.err.println(clazz);
			}
			catch (Exception ex) {
				throw new RuntimeException("Failed to load class %s".formatted(className), ex);
				// ex.printStackTrace(System.err);
			}
		});

		System.err.println("------------------------------------------------");

		List<URI> resources = streamResolvedModules(Predicate.isEqual(moduleName))//
				.map(ResolvedModule::reference)//
				.map(moduleReference -> scanForResources(moduleReference, "^com/example/.+\\.class$"))//
				.flatMap(List::stream)//
				.toList();

		resources.forEach(resource -> {
			try {
				System.err.println(resource.toURL());
			}
			catch (MalformedURLException ex) {
				ex.printStackTrace();
			}
		});
	}

	private Stream<ResolvedModule> streamResolvedModules(Predicate<String> moduleNamePredicate) {
		ModuleLayer layer = getClass().getModule().getLayer();
		if (layer == null) {
			layer = ModuleLayer.boot();
		}
		return layer.configuration().modules().stream()//
				.filter(module -> moduleNamePredicate.test(module.name()));
	}

	private List<String> scanForNames(ModuleReference reference, String regex) {
		try (ModuleReader reader = reference.open()) {
			try (Stream<String> names = reader.list()) {
				return names.filter(name -> name.matches(regex)).toList();
			}
		}
		catch (IOException ex) {
			throw new UncheckedIOException("Failed to read contents of " + reference, ex);
		}
	}

	private List<URI> scanForResources(ModuleReference reference, String regex) {
		try (ModuleReader reader = reference.open()) {
			try (Stream<String> names = reader.list()) {
				return names.filter(name -> name.matches(regex))//
						.map(name -> {
							try {
								return reader.find(name);
							}
							catch (IOException ex) {
								ex.printStackTrace();
							}
							return Optional.<URI> empty();
						})//
						.filter(Optional::isPresent)//
						.map(Optional::get).toList();
			}
		}
		catch (IOException ex) {
			throw new UncheckedIOException("Failed to read contents of " + reference, ex);
		}
	}

}

The output of running that is:

[file [/Users/sbrannen/source/spring-issues/module-path-scanning/target/test-classes/com/example/ScanningTest.class]]
------------------------------------------------
com/example/One.class
com/example/ScanningTest.class
------------------------------------------------
class com.example.One
class com.example.ScanningTest
------------------------------------------------
file:/Users/sbrannen/source/spring-issues/module-path-scanning/target/classes/com/example/One.class
file:/Users/sbrannen/source/spring-issues/module-path-scanning/target/test-classes/com/example/ScanningTest.class

The first output demonstrates the issue: PathMatchingResourcePatternResolver only finds ScanningTest.class and not One.class.

The final output demonstrates that we can find resources (as instances of URI) using the Module APIs and successfully find both ScanningTest.class and One.class.

Kudos to @sormuras for providing inspirational use of the java.lang.module APIs in JUnit 5!


Where to go from here?

We may introduce a new ModulePathResource as a companion to ClassPathResource, where the logic in ModulePathResource is similar to the logic in the above proof of concept.

We may then introduce additional support in PathMatchingResourcePatternResolver that resolves ModulePathResource instances when appropriate.

sbrannen added a commit to sbrannen/spring-framework that referenced this issue May 22, 2022
Prior to this commit, searching for classpath resources using the
"classpath*:" resource prefix did not find all applicable resources
in a patched module -- for example, when test classes and resources are
patched into the application module using Maven Surefire.

This affected component scanning -- for example, via @componentscan --
and PathMatchingResourcePatternResolver.getResources(String) in general.

This commit addresses this by introducing first-class support for
scanning the module path when PathMatchingResourcePatternResolver's
getResources(String) method is invoked with a location pattern using
the "classpath*:" resource prefix. If resources are resolved via the
module path, the classpath will not be searched; otherwise, the existing
classpath scanning algorithm will be used.

See spring-projectsgh-21515
@sbrannen
Copy link
Member

sbrannen commented May 22, 2022

Current work on this issue can be viewed in the following feature branch.

https://github.com/sbrannen/spring-framework/commits/module-path-scanning

sbrannen added a commit to sbrannen/spring-framework that referenced this issue May 23, 2022
Prior to this commit, searching for classpath resources using the
"classpath*:" resource prefix did not find all applicable resources
in a patched module -- for example, when test classes and resources are
patched into the application module using Maven Surefire.

This affected component scanning -- for example, via @componentscan --
and PathMatchingResourcePatternResolver.getResources(String) in general.

This commit addresses this by introducing first-class support for
scanning the module path when PathMatchingResourcePatternResolver's
getResources(String) method is invoked with a location pattern using
the "classpath*:" resource prefix. If resources are resolved via the
module path, the classpath will not be searched; otherwise, the existing
classpath scanning algorithm will be used.

See spring-projectsgh-21515
sbrannen added a commit to sbrannen/spring-framework that referenced this issue May 23, 2022
Prior to this commit, searching for classpath resources using the
"classpath*:" resource prefix did not find all applicable resources for
applications deployed as modules -- for example, when test classes and
resources are patched into the application module automatically by
Maven Surefire.

This affected component scanning -- for example, via [@]ComponentScan --
and PathMatchingResourcePatternResolver.getResources(String) in
general.

This commit addresses this by introducing first-class support for
scanning the module path when PathMatchingResourcePatternResolver's
getResources(String) method is invoked with a location pattern using
the "classpath*:" resource prefix. Specifically, getResources(String)
first searches all modules in the boot layer, excluding system modules.
It then searches the classpath using the existing Classloader-based
algorithm and returns the combined results.

See spring-projectsgh-21515
@sbrannen
Copy link
Member

sbrannen commented May 23, 2022

Hi everybody,

Thanks for all of the feedback over the years!

I am now closing this issue since it has been superseded by the following issues.

Please follow those issues for additional updates on module path scanning and resource support.

@sbrannen sbrannen removed this from the 6.0.0-M5 milestone May 23, 2022
@sbrannen sbrannen added status: superseded An issue that has been superseded by another and removed type: bug A general bug labels May 23, 2022
@sbrannen
Copy link
Member

sbrannen commented May 24, 2022

If you've been following this issue, I'm happy to let you know that #28506 has been resolved for inclusion in 6.0 M5.

In the interim, feel free to try out 6.0 snapshots with modular testing using Maven.

I've created a new repository for demonstrating the use of the Spring Framework with the Java Module System: https://github.com/sbrannen/spring-module-system

That repository currently contains a maven-surefire-patched-module project which demonstrates support for @ComponentScan in a patched module using Maven Surefire.

dkfellows added a commit to SpiNNakerManchester/JavaSpiNNaker that referenced this issue Aug 18, 2022
I believe the issue in spring-projects/spring-framework#21515 affects
this code, and it means that I can't yet recommend adopting this PR as
we currently fail all sorts of tests because classpath scanning just
collapses completely after module-isation. The workaround is to run
tests with the module path disabled, but at least Eclipse seems
completely unable to do anything useful with that; you get a total
failure to run tests. This is an abysmal mess!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in: core Issues in core modules (aop, beans, core, context, expression) in: test Issues in the test module status: superseded An issue that has been superseded by another
Projects
None yet
Development

No branches or pull requests