Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ClassLoader Leak using TestProfiles (OutOfMemoryError: Metaspace) #38774

Open
xuckz opened this issue Feb 14, 2024 · 7 comments
Open

ClassLoader Leak using TestProfiles (OutOfMemoryError: Metaspace) #38774

xuckz opened this issue Feb 14, 2024 · 7 comments
Labels
area/testing kind/bug Something isn't working

Comments

@xuckz
Copy link

xuckz commented Feb 14, 2024

Describe the bug

Using multiple TestProfiles results in OutOfMemoryError: Metaspace
It seems that running a @QuarkusTest with its own @\TestProfile is not cleaned up properly after the Test has finished, and all QuarkusClassLoaders used are permanently stored in MetaSpace.

Each Test adding its own ClassLoader to MetaSpace until the OutOfMemoryError occurs.

Increasing the MetaSpace for each @QuarkusTest with its own @\TestProfile is a limited workaround.

Edit: I think this might be related to #12498 since the Heap Space is also effected.

Edit2: Screenshot added

image-2024-02-13-13-01-49-831

Expected behavior

The expected behavior would be, that you are not limited to any amount of @\TestProfil and that each @\TestProfile would not increase the amount of MetaSpace required.

Actual behavior

Each @QuarkusTest with its own @\TestProfile requires the MetaSpace increased.

How to Reproduce?

Ready to go repo:
https://github.com/xuckz/quarkus-quickstarts-classloader-leak-demo
xuckz/quarkus-quickstarts-classloader-leak-demo@ccfa144

I used the quarkus-quickstarts/hibernate-orm-panache-quickstart as showcase:

  • Checkout https://github.com/quarkusio/quarkus-quickstarts
  • Open hibernate-orm-panache-quickstart/pom.xml
  • Limit MetaSpace (Since the amount of classes loaded by the quickstart examples is very low, we will have to cap the MetaSpace manually, otherwise we would need a lot of more tests)
<plugin>
    <!-- you need this specific version to integrate with the other build helpers -->
    <artifactId>maven-surefire-plugin</artifactId>
    <version>${surefire-plugin.version}</version>
    <configuration>
        <argLine>-XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=256M</argLine>
        <systemPropertyVariables>
            <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
            <maven.home>${maven.home}</maven.home>
        </systemPropertyVariables>
    </configuration>
</plugin>
  • Add 1 empty Test class
package org.acme.hibernate.orm.panache.test;

import org.junit.jupiter.api.Test;

import io.quarkus.test.junit.QuarkusTest;
import io.quarkus.test.junit.QuarkusTestProfile;
import io.quarkus.test.junit.TestProfile;

@QuarkusTest
@TestProfile(LeakHunt1Test.LeakHuntTestProfile.class)
public class LeakHunt1Test {

    public static class LeakHuntTestProfile implements QuarkusTestProfile {
    }

    @Test
    void test() {
        System.out.println("asdfasdf");
    }
}
  • Run mvn test -> Everything runs fine
  • Copy LeakHunt1Test.java 9 more times
  • Run mvn test -> MetaSpace error

Output of uname -a or ver

Linux xxx 5.4.0-167-generic #184-Ubuntu SMP Tue Oct 31 09:21:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Output of java -version

openjdk version "17.0.9" 2023-10-17 OpenJDK Runtime Environment (build 17.0.9+9-Ubuntu-120.04) OpenJDK 64-Bit Server VM (build 17.0.9+9-Ubuntu-120.04, mixed mode, sharing)

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.6.3 Maven home: /usr/share/maven Java version: 17.0.9, vendor: Private Build, runtime: /usr/lib/jvm/java-17-openjdk-amd64 Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "5.4.0-167-generic", arch: "amd64", family: "unix"

Additional information

No response

@xuckz xuckz added the kind/bug Something isn't working label Feb 14, 2024
@quarkus-bot
Copy link

quarkus-bot bot commented Feb 15, 2024

/cc @geoand (testing)

@Adelrisk
Copy link

I don't want to be a "I'm having this problem too" commentator, but ... I'm having this problem too. But with only three profiles, each applied to a different test case/test class.

Just for an idea of the scope: real-world project, kotlin and gradle, two entities, few endpoints, 23 passing tests and good coverage (this service is very CRUD-like). (A useful utility microservice that does a single, small job well.)

I then introduced three test profiles that configure two properties differently, such as:

class NoSsiIssuanceV1Profile: QuarkusTestProfile {
    override fun getConfigOverrides(): Map<String, String> {
        return mapOf(
            "quarkus.rest-client.ssi-issuance-v1-api.url" to "",
            "quarkus.rest-client.ssi-issuance-v2-api.url" to "http://ssi-v1:8080"
        )
    }
}

I use each profile once to verify the behaviour of injection I programmed into the application, such as:

@QuarkusTest
@TestProfile(NoSsiIssuanceProfile::class)
open class NoSsiClientTest {

    @Inject
    lateinit var sut: Set<ApiVersion>
    @Inject
    lateinit var v1: Optional<SsiIssuanceV1HttpClient>
    @Inject
    lateinit var v2: Optional<SsiIssuanceV2HttpClient>

    @Test
    fun hasCorrectApiVersions() {
        assertThat(sut, equalTo(setOf()))
    }
    @Test
    fun hasNoV1Client() {
        assertThat(v1.isEmpty, equalTo(true))
    }
    @Test
    fun hasNoV2Client() {
        assertThat(v2.isEmpty, equalTo(true))
    }
}

When I run all my tests, the JVM prematurely aborts the execution of all three test classes marked with @TestProfile. As an experiment, I set a filter so only 10 other tests were executed, in which case "only" one test class with a test profile aborted due to OOM.

This gives the impression that the complexity threshold is quite low before this bug leads to an OutOfMemoryError.

@geoand
Copy link
Contributor

geoand commented Mar 15, 2024

Can you attach your sample application that leads to the OOME?

@Guillaume-Lebegue
Copy link

I also have that problem on my project
We use around 10 different profiles using multiple databases, running Kafka or a scheduler.
And today when I added a new one my CI crashed with an OOME.

After checking, I noticed that each new Profile created a jump in memory use.
And by the last test, the JVM was using 6G of RAM, which is my runner's limit, so OOME.

I made a quick repo to show the problem https://github.com/Guillaume-Lebegue/quarkus_test_out_of_memory_testprofile
In it, 10 tests, each with its profile, and each profile and test are doing the same thing.

When running the tests, the first test ended with ~500Mo of memory used, and the last test was at 1200Mo of memory used.
I also created another branch only_one_profile that makes all 10 tests run on the same profile; this time, it did not go more than 500Mo.

@janek64
Copy link

janek64 commented Apr 15, 2024

I am facing the same issue with only two test profiles. All tests of the first profile are running as expected, but the startup of the second profile fails with an OutOfMemoryError.

A new observation that I can add is that the issue only occurs since I added io.quarkus:quarkus-jacoco to my project. When I remove the dependency, all tests are running as expected.

@billapepper
Copy link

billapepper commented May 30, 2024

Another "me too", but I recently started migrating from 3.2.x to 3.8.4 (as the LTS is expiring on 3.2.x in July).

I ran into a problem where @InjectSpy began throwing a "Please don't pass mock here. Spy is not allowed on mock." error when trying to spy using a smallrye config... this seemingly stems from the Mockito upgrade (https://github.com/mockito/mockito/pull/2989/files)... but I couldn't find a workaround, so instead, I reworked my application to use @TestProfile and override the configuration for the specific tests. Very basic profiles, in a similar fashion to how #38774 (comment) did:

    public static class PublicPrivateDomains implements QuarkusTestProfile {


        @Override
        public Map<String, String> getConfigOverrides() {
            return Map.of(
                    "prime.landing-service.public-domains", ".domain.com",
                    "prime.landing-service.private-domains", ".domain.net"
            );
        }
    }

However, since doing this, the build with several of these tests fails with a OOM error. I have tried increasing the memory using <argLine>-XX:MetaspaceSize=256M -XX:MaxMetaspaceSize=2048M -XX:MaxRAMPercentage=75</argLine> on the maven-surefire-plugin, but it still runs out of memory in our GH actions.

These tests ran without issues when they were not using separate @TestProfiles.

Error:  org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process
Error:  Metaspace
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:628)
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:285)
Error:  	at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250)
Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1203)
Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1055)
Error:  	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:871)
Error:  	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:370)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:351)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:215)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:***1)
Error:  	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:163)
Error:  	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:1***)
Error:  	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
Error:  	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
Error:  	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
Error:  	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:299)
Error:  	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
Error:  	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
Error:  	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:963)
Error:  	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:296)
Error:  	at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
Error:  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Error:  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
Error:  	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)

EDIT: Also, removing quarkus-jacoco did not seem to stop the problem for me (with regards to #38774 (comment))

@aylwyne
Copy link

aylwyne commented Jun 11, 2024

I'm wondering if there is any traction on this issue. This is blocking our ability to upgrade to 3.8.4 because our test suite uses test profiles quite a lot and we haven't been able to get any alternatives working (see previous comment from billapepper). We are currently on the previous LTS version (3.2.x) which is expiring soon yet are blocked on upgrading to the 3.8.x LTS release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/testing kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

8 participants