Recreate ConfigurableApplicationContext may cause OutOfMemory when using spring-boot-starter-actuator

[qfdk]

I have a SpringBoot application using actuator, in my method update Context, I need to recreate ConfigurableApplicationContext in order to use the new properties updated in application.properties, because in SpringBoot actuator doesn’t have “refresh” endpoint.

In production this service may called only 1 time by week. After 3 mouths I got OOM.

After analyzing of headdump in prod, the problem came form io.micrometer using by actuator.

io.micrometer.core.instrument.binder.tomcat.TomcatMetrics

I created a demo project to isole le problem.

pom.xml

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.0.7.RELEASE</version>
        <!--same problem-->
        <!--<version>2.2.2.RELEASE</version>-->
        <relativePath/> <!-- lookup parent from repository -->
    </parent>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.8</version>
            <scope>provided</scope>
        </dependency>
    </dependencies>

DemoApplication

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.scheduling.annotation.EnableScheduling;

@SpringBootApplication
@EnableScheduling
public class DemoApplication {

    public static ConfigurableApplicationContext context = null;

    public static void main(String[] args) {
        DemoApplication.context = SpringApplication.run(DemoApplication.class, args);
    }

}

MyController

package me.qfdk.demo;

import lombok.extern.slf4j.Slf4j;
import org.springframework.boot.SpringApplication;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.concurrent.ArrayBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

@RestController
@Slf4j
public class MyController {
    @GetMapping("/")
    // call every 5 s
    @Scheduled(cron = "0/5 * * * * ?")
    public void test() {
        ThreadPoolExecutor threadPool;
        threadPool = new ThreadPoolExecutor(1, 1, 0, TimeUnit.SECONDS, new ArrayBlockingQueue<>(1), new ThreadPoolExecutor.DiscardOldestPolicy());
        threadPool.execute(() -> {
            // close context
            DemoApplication.context.close();
            log.info("[update] create new Context");
            DemoApplication.context = SpringApplication.run(DemoApplication.class, "");
            log.info("[update] update done");
        });
        threadPool.shutdown();
    }
}

using JVM option to give the problem faster -Xms128m -Xmx128m -XX:+HeapDumpOnOutOfMemoryError

Demo to download: http://qfdk.me/demo.zip

AutoCloseable may be to implement in TomcatMetrics

[wilkinsona]

Thanks for the report and the sample. I believe the problem is that TomcatMetrics only removes its NotificationListener inner-class when the MBean it's looking for becomes available. If that never happens, the listener will never be removed. This will have to be addressed in Micrometer. Can you please open an issue over there and comment here with a link to it?

[qfdk]

Thank for your replay. I create a ticket there
micrometer-metrics/micrometer#1759