The metrics on the jvm do not appear when I call actuator/prometheus

[patbaie974]

I have REST APIs that use SPRING and for which my metrics on the JVM appear well, such as :
2 "jvm.buffer.count"
3 "jvm.buffer.memory.used"
4 "jvm.buffer.total.capacity"
5 "jvm.classes.loaded"
6 "jvm.classes.unloaded"
7 "jvm.gc.live.data.size"
8 "jvm.gc.max.data.size"
9 "jvm.gc.memory.allocated"
10 "jvm.gc.memory.promoted"
11 "jvm.gc.overhead"
12 "jvm.gc.pause"
13 "jvm.memory.committed"
14 "jvm.memory.max"
15 "jvm.memory.usage.after.gc"
16 "jvm.memory.used"
17 "jvm.threads.daemon"
18 "jvm.threads.live"
19 "jvm.threads.peak"
20 "jvm.threads.states"

On the other hand, I have other REST APIs that make calls to other APIs via org.springframework.web.reactive.function.client.WebClient and now I can't see the metrics on the jvm. On the other hand I have metrics on executor and tomcat :
2 "executor.active"
3 "executor.completed
4 "executor.pool.core"
5 "executor.pool.max"
6 "executor.pool.size"
7 "executor.queue.remaining"
8 "executor.queued
9 "tomcat.sessions.active.current"
10 "tomcat.sessions.active.max"
11 "tomcat.sessions.alive.max"
12 "tomcat.sessions.created"
13 "tomcat.sessions.expired"
14 "tomcat.sessions.rejected"

I can't figure out why this difference in behaviour.

Thanks for your help

[checketts]

You'll want to add the SpringBoot/JVM/config details. Also this probably would be better as a StackOverflow question...

[jonatan-ivanov]

When you opened the issue, the template asked you additional details like what's your environment:

• Micrometer version
• Micrometer registry
• OS
• Java version
  of how to reproduce the issue. Could you please add those too if this is still an issue?
  Other things you can check:
• Please use a version that is supported (latest 1.9.x, 1.10.x, or 1.11.x). You can also take a look at our samples where we have Spring Boot apps that make these metrics available.
• Please check if Spring Boot configures the necessary binders.
• Please make sure GC was triggered if you are looking for GC metrics (otherwise you will see no GC metrics).
• Please check if you are not using the global registry and Boot registers the binders to the same registry that publishes the data.

If it still not work, could you please provide a minimal java project to reproduce this issue so we can more easily investigate and ensure any fix is working properly for your use case?
In the meantime let me close this issue, please get back to us if you have a reproducer and want us to reopen.

[patbaie974]

Hi Jonatan, thanks a lot for your back et sorry for sorry for the lack of detail in the issue.
For your information.

I have a JAVA/SPRING API that uses a common library broken down into several modules that instantiate the Actuator/metrics part, the security module for our APIs and a module that also instantiates the WebClients associated with the APIs we consume, and it's obviously this part that impacts the metrics returned in acutator/prometheus.

If I don't add the API Consumer module, and therefore the WebClients, my API returns the metrics perfectly, including JVM, GC, etc. metrics.

As soon as I add the Consumer API module, I no longer see the metrics on the JVM, GC, but I continue to see the metrics on the calls to my endPoints.

It's a bit complicated to detail the code of this Consumer API module, because it's used by our company or privately.

Versions used :
<java.version>17</java.version>
<maven.compiler.source>17</maven.compiler.source>
<maven.compiler.target>17</maven.compiler.target>
<maven-compiler-plugin.version>3.8.1</maven-compiler-plugin.version>
<commons-lang3.version>3.12.0</commons-lang3.version>
<spring-boot.version>2.7.0</spring-boot.version>
<spring-security.version>5.7.1</spring-security.version>
<webjars-locator.version>0.45</webjars-locator.version>

  <dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-core</artifactId>
    <version>1.9.0</version>
  </dependency>

1/ In the Actuator Autoconfigure module I have :
POM :


org.springframework.boot
spring-boot-starter-actuator


io.micrometer
micrometer-registry-prometheus

And I load the next properties with the org.springframework.boot.env.EnvironmentPostProcessor=re.xxx.api.actuator.LoadActuatorDefaultProperties

management.server.port=4242
management.endpoint.health.show-details=always
management.endpoints.web.exposure.include=*

management.endpoints.jmx.exposure.exclude=*
spring.jmx.enabled=true

management.endpoint.health.probes.enabled=true
management.endpoint.shutdown.enabled=false
management.cloudfoundry.enabled=false
management.endpoint.jolokia.enabled=false
management.metrics.export.appoptics.enabled=false
management.metrics.export.atlas.enabled=false
management.metrics.export.datadog.enabled=false
management.metrics.export.dynatrace.enabled=false
management.metrics.export.elastic.enabled=false
management.metrics.export.ganglia.enabled=false
management.metrics.export.graphite.enabled=false
management.metrics.export.humio.enabled=false
management.metrics.export.influx.enabled=false
management.metrics.export.jmx.enabled=false
management.metrics.export.kairos.enabled=false
management.metrics.export.newrelic.enabled=false
management.metrics.export.signalfx.enabled=false
management.metrics.export.stackdriver.enabled=false
management.metrics.export.statsd.enabled=false
management.metrics.export.wavefront.enabled=false

2/ In the API Consumer Module I load the next JAVA class
re.xxx.consumer.JavaNetSSLContextAutoConfiguration
re.xxx.consumer.NettyCustomizationAutoConfiguration
re.xxx.consumer.WebClientInstanceBuilderAutoConfiguration
re.xxx.consumer.oauth.OAuth2WebClientInstanceBuilderAutoConfiguration
re.xxx.consumer.oauth.ReactiveOAuth2ClientCredentialsAutoConfiguration

In the WebClientInstanceBuilderAutoConfiguration I have :

import org.springframework.boot.autoconfigure.AutoConfiguration;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Conditional;
import org.springframework.web.reactive.function.client.WebClient;

@autoConfiguration
@EnableConfigurationProperties({ ConsumerProperties.class })
@ConditionalOnClass(WebClient.class)
@ConditionalOnBean(WebClient.Builder.class)
public class WebClientInstanceBuilderAutoConfiguration {
@bean
@conditional(ServiceConfiguredCondition.class)
@ConditionalOnMissingBean
public WebClientInstanceBuilder webClientInstanceBuilder(ConsumerProperties consumerProperties,
WebClient.Builder webClientBuilder) {

	return new WebClientInstanceBuilder(consumerProperties, webClientBuilder);
}

}

In the ServiceConfiguredCondition I load de properties for the API that we want to use
public class ServiceConfiguredCondition extends SpringBootCondition {

private static final Bindable<Map<String, ConsumerProperties.ServiceProperties>> STRING_REGISTRATION_MAP = Bindable
		.mapOf(String.class, ConsumerProperties.ServiceProperties.class);

@Override
public ConditionOutcome getMatchOutcome(ConditionContext context, AnnotatedTypeMetadata metadata) {
	ConditionMessage.Builder message = ConditionMessage.forCondition("Consumer Services configured condition");
	Map<String, ConsumerProperties.ServiceProperties> registrations = getRegistrations(context.getEnvironment());

	if (!registrations.isEmpty()) {
		return ConditionOutcome.match(message.foundExactly("configured services c" + registrations.values().stream()
				.map(ConsumerProperties.ServiceProperties::toString).collect(Collectors.joining(", "))));
	} else {
		return ConditionOutcome.noMatch(message.notAvailable("configured service"));

	}
}

private Map<String, ConsumerProperties.ServiceProperties> getRegistrations(Environment environment) {
	return Binder.get(environment).bind("xoapi.consumer.services", STRING_REGISTRATION_MAP)
			.orElse(Collections.emptyMap());
}

}

In the WebClientInstanceBuilder I have :

public class WebClientInstanceBuilder implements BeanDefinitionRegistryPostProcessor {
ConsumerProperties consumerProperties;
WebClient.Builder webClientBuilder;
Long maxInMemorySize;

public WebClientInstanceBuilder(ConsumerProperties properties, WebClient.Builder webClientBuilder) {
	this.consumerProperties = properties;
	this.maxInMemorySize    = properties.getMaxInMemorySize().toBytes();
	this.webClientBuilder   = webClientBuilder;
}

protected WebClient.Builder customizeBuilder(String key, WebClient.Builder serviceBuilder, ServiceProperties service) {
	return serviceBuilder.baseUrl(service.getUrl()).defaultHeaders(httpHeaders -> {
		service.getHeaders().forEach(httpHeaders::add);
	}).codecs( codecs -> {
		codecs.defaultCodecs().maxInMemorySize(this.maxInMemorySize.intValue());
	});
}

protected String webClientBeanName(String key, ServiceProperties service) {
	return String.format("%sWebClient", key);
}

@Override
public void postProcessBeanDefinitionRegistry(BeanDefinitionRegistry registry) throws BeansException {

	this.consumerProperties.getServices().forEach((key, service) -> {
		String beanKey = webClientBeanName(key, service);
		
		WebClient client = customizeBuilder(beanKey, webClientBuilder.clone(), service).build();

		GenericBeanDefinition definition = new GenericBeanDefinition();
		definition.setBeanClass(WebClient.class);
		definition.setInstanceSupplier(() -> client);
		registry.registerBeanDefinition(beanKey, definition);

		logger.info("Bean {} has been configured to access to {}", beanKey, service.getUrl());
	});
}

@Override
public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) throws BeansException {
}

private static final Logger logger = LoggerFactory.getLogger(WebClientInstanceBuilder.class);

}

Maybe the problem comes from the registry like you said in the issue => Please check if you are not using the global registry and Boot registers the binders to the same registry that publishes the data

Maybe if you want more information I can send you by mail ?

[jonatan-ivanov]

When I asked for a minimal java project, what I meant is really a sample project, e.g.: in a GitHub repo that I can clone and start that reproduces the issue.
The above comment is neither a project (it's kind of instructions how to create one), nor minimal (is OAuth really needed? also, all of those properties are necessary? with all the custom configuration?) nor compiling if I would copy it into a project.

I would like to help you but at the same time I'm not very keen on spending a day (or days) on trying to figure out what you are trying to do in the comment above.
This is a good example for such a reproducer created by somebody else for another issue: https://github.com/davidmelia/spring-boot-kafka-consumer-tracing/tree/boot_3.0.9_trace_issue

Here's a good description about how to create one: https://stackoverflow.com/help/minimal-reproducible-example

[patbaie974]

Hello, I understand. I'll try to create a example but could you please just give me more information about your first return : => Please check if you are not using the global registry and Boot registers the binders to the same registry that publishes the data Because I think that it's the problem. What do you mean by global registry? Thanks a lot Regards Patrick Le jeu. 17 août 2023 à 10:17, Jonatan Ivanov ***@***.***> a écrit :

…

When I asked for a minimal java project, what I meant is really a sample project, e.g.: in a GitHub repo that I can clone and start that reproduces the issue. The above comment is neither a project (it's kind of instructions how to create one), nor minimal (is OAuth really needed? also, all of those properties are necessary? with all the custom configuration?) nor compiling if I would copy it into a project. I would like to help you but at the same time I'm not very keen on spending a day (or days) on trying to figure out what you are trying to do in the comment above. This is a good example for such a reproducer created by somebody else for another issue: https://github.com/davidmelia/spring-boot-kafka-consumer-tracing/tree/boot_3.0.9_trace_issue — Reply to this email directly, view it on GitHub <#4011 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AX36RFUEBWJHLWWJP7S4HDTXVWZPZANCNFSM6AAAAAA3JX636I> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jonatan-ivanov]

See the docs for the global registry: https://micrometer.io/docs/concepts#_global_registry
It basically means that you should not use Metrics.globalRegistry but you should inject the MeterRegistry into your components, something like this:

public class MyComponent {
    private final MeterRegistry registry;

    public MyComponent(MeterRegistry registry) {
        this.registry = registry;
    }
}

[patbaie974]

I'm not sure I understand, because for me it's automatic via the following dependencies: <dependencies> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> </dependency> </dependencies> I don't have any specific code to manage metrics Le ven. 18 août 2023 à 09:19, Jonatan Ivanov ***@***.***> a écrit :

…

See the docs for the global registry: https://micrometer.io/docs/concepts#_global_registry It basically means that you should not use Metrics.globalRegistry but you should inject the MeterRegistry into your components, something like this: public class MyComponent { private final MeterRegistry registry; public MyComponent(MeterRegistry registry) { this.registry = registry; } } — Reply to this email directly, view it on GitHub <#4011 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AX36RFQRT24AK4ZJODKZI3DXV33PFANCNFSM6AAAAAA3JX636I> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jonatan-ivanov]

If you don't have any specific code to manage metrics, then I don't understand your comment above: #4011 (comment)

If you just use Boot and Micrometer, you should see the JVM metrics. Please check out our samples as they show them or please provide a minimal java project to reproduce this issue so we can more easily investigate.

[patbaie974]

Hi Jonatan,

Here you can find 2 projetcs for the test and your analyse :

1/ lib-api-common : is a librairy that we use in the pointsofsale-api
To instal the lib : mvn clean install

2/ pointsofsale-api : just a API with one get endPoint for the test and wich use lib-api-common

• In the pointsofsale-api\src\main\resources\application.properties, as you can see the following lines are commented :

#xoapi.consumer.services.accountManagement.url=https://test.api:8443/test
#xoapi.consumer.services.accountManagement.headers[Content-Type]=application/json
#xoapi.consumer.services.accountManagement.headers[Accept-Language]=fr
#xoapi.consumer.services.accountManagement.oauth2-client-registration-id=pos

OAuth2 Client configuration for Comarch service

#spring.security.oauth2.client.registration.pos.client-id=test
#spring.security.oauth2.client.registration.pos.client-secret=test
#spring.security.oauth2.client.registration.pos.authorization-grant-type=client_credentials
#spring.security.oauth2.client.registration.pos.scope=
#spring.security.oauth2.client.registration.pos.provider=monprovider

OAuth2 AS Kong (for COMARCH)

#spring.security.oauth2.client.provider.monprovider.token-uri=https://test:8443/auth/oauth2/token

#enable.insecuredtrustmanager=true

These lines are used to declare an endPoint to be used within our API. These lines tell the common lib, and in particular the API CONSUMER part (devfactory-api-consumer-autoconfigure), to instantiate a WebClient with the characteristics described in the properties. See ConsumerProperties.java, WebClientInstanceBuilderAutoConfiguration.java and WebClientInstanceBuilder.java.
So as these lines are commented the lib common doesn't instantiate the Webclient.

• To install the api : mvn clean install

3/ Run the API pointsofsale-api : java -jar target\pos-api-1.0.1.jar

4/ Test the metrics with http://localhost:4242/actuator/prometheus and I can see the metrics on the jvm and other

HELP jvm_gc_memory_promoted_bytes_total Count of positive increases in the size of the old generation memory pool before GC to after GC

TYPE jvm_gc_memory_promoted_bytes_total counter

jvm_gc_memory_promoted_bytes_total 6.764032E7

HELP jvm_gc_live_data_size_bytes Size of long-lived heap memory pool after reclamation

TYPE jvm_gc_live_data_size_bytes gauge

jvm_gc_live_data_size_bytes 0.0

HELP jvm_classes_unloaded_classes_total The total number of classes unloaded since the Java virtual machine has started execution

TYPE jvm_classes_unloaded_classes_total counter

jvm_classes_unloaded_classes_total 0.0

5/ In the pointsofsale-api\src\main\resources\application.properties, the following lines must be uncommented :

xoapi.consumer.services.accountManagement.url=https://test.api:8443/test
xoapi.consumer.services.accountManagement.headers[Content-Type]=application/json
xoapi.consumer.services.accountManagement.headers[Accept-Language]=fr
xoapi.consumer.services.accountManagement.oauth2-client-registration-id=pos

OAuth2 Client configuration for Comarch service

spring.security.oauth2.client.registration.pos.client-id=test
spring.security.oauth2.client.registration.pos.client-secret=test
spring.security.oauth2.client.registration.pos.authorization-grant-type=client_credentials
spring.security.oauth2.client.registration.pos.scope=
spring.security.oauth2.client.registration.pos.provider=monprovider

OAuth2 AS Kong (for COMARCH)

spring.security.oauth2.client.provider.monprovider.token-uri=https://test:8443/auth/oauth2/token

enable.insecuredtrustmanager=true

So as these lines are not any more commented the lib common does instantiate the Webclient to use AccountManagement.

6/ Re build de pointsofsale-api => mvn clean install

7/ Run the API pointsofsale-api : java -jar target\pos-api-1.0.1.jar

8/ Test the metrics with http://localhost:4242/actuator/prometheus and I can't see the metrics on the jvm

HELP tomcat_sessions_active_max_sessions

TYPE tomcat_sessions_active_max_sessions gauge

tomcat_sessions_active_max_sessions 0.0

HELP executor_queue_remaining_tasks The number of additional elements that this queue can ideally accept without blocking

TYPE executor_queue_remaining_tasks gauge

executor_queue_remaining_tasks{name="applicationTaskExecutor",} 2.147483647E9
executor_queue_remaining_tasks{name="statusBuilderExecutor",} 2.147483647E9

SO, when I use the devfactory-api-consumer-autoconfigure to instantiate the WebClient that I want to use in my PointsOfSale API I don't have anymore the JVM's metrics.
Maybe the problem is that in the lib common we use the BeanDefinitionRegistry ?

pointsofsale-api.zip
lib-api-commons.zip

Thanks a lot

[patbaie974]

Hi, I put in #4011 the code to test on your side. Thanks for your analysis. Regards Patrick Le ven. 18 août 2023 à 21:35, Jonatan Ivanov ***@***.***> a écrit :

…

If you don't have any specific code to manage metrics, then I don't understand your comment above: #4011 (comment) <#4011 (comment)> If you just use Boot and Micrometer, you should see the JVM metrics. Please check out our samples as they show them or please provide a minimal java project to reproduce this issue so we can more easily investigate. — Reply to this email directly, view it on GitHub <#4011 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AX36RFV2NOJUVHXKWK67RUTXV6RWDANCNFSM6AAAAAA3JX636I> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[patbaie974]

Hi Jonatan, Do you need some other information for your investigation? BR Patrick Le ven. 18 août 2023 à 21:35, Jonatan Ivanov ***@***.***> a écrit :

…

If you don't have any specific code to manage metrics, then I don't understand your comment above: #4011 (comment) <#4011 (comment)> If you just use Boot and Micrometer, you should see the JVM metrics. Please check out our samples as they show them or please provide a minimal java project to reproduce this issue so we can more easily investigate. — Reply to this email directly, view it on GitHub <#4011 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AX36RFV2NOJUVHXKWK67RUTXV6RWDANCNFSM6AAAAAA3JX636I> . You are receiving this because you authored the thread.Message ID: ***@***.***>