Add Jetty's ThreadPool metrics

[matsumana]

I want to monitor whether Jetty's ThreadPool setting is sufficient so I'd like to add several metrics.

Usage

Example In Spring Boot 2.0.x

@Bean
public JettyServletWebServerFactory webServerFactory(MeterRegistry registry) {
    JettyServletWebServerFactory factory = new JettyServletWebServerFactory();
    tags = Collections.singletonList(Tag.of("id", "0"));
    factory.setThreadPool(new InstrumentedQueuedThreadPool(registry, tags));
    return factory;
}

[checketts]

How does this compare to the Tomcat equivalent? Wondering if active is a better name and how it differs from busy? Or is it merely idle + busy?

[matsumana]

I decided the name live with reference to JvmThreadMetrics.
https://github.com/micrometer-metrics/micrometer/blob/v1.0.3/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/jvm/JvmThreadMetrics.java#L57

For the meaning of the metric please refer to the Jetty's source.
https://github.com/eclipse/jetty.project/blob/jetty-9.4.8.v20171121/jetty-util/src/main/java/org/eclipse/jetty/util/thread/QueuedThreadPool.java#L428-L455

[matsumana]

Unified with Tomcat's metrics name.
(from live to current)
https://github.com/micrometer-metrics/micrometer/blob/v1.0.3/micrometer-core/src/main/java/io/micrometer/core/instrument/binder/tomcat/TomcatMetrics.java#L89

[matsumana]

Hello.
Would you please review this?

[mweirauch]

Just a thought: Is the jetty.threads.idle superfluous as it could be calculated from jetty.threads.current subtracted by jetty.threads.busy? Wonder who would graph the "idle" metric in a plot with "busy" and "current" and "max".

[matsumana]

Just a thought: Is the jetty.threads.idle superfluous as it could be calculated from jetty.threads.current subtracted by jetty.threads.busy?

@mweirauch

Good point.
I'll delete jetty.threads.idle and unify metric name to the same it as TomcatMetrics.

• jetty.threads.config.min
• jetty.threads.config.max
• jetty.threads.busy
• jetty.threads.current

[matsumana]

Updated

[mweirauch]

Just tested against Spring Boot 1.5 with

	@Bean
	public JettyEmbeddedServletContainerFactory webServerFactory(MeterRegistry registry) {
		JettyEmbeddedServletContainerFactory factory = new JettyEmbeddedServletContainerFactory();
		factory.setThreadPool(new InstrumentedQueuedThreadPool(registry, Collections.emptyList()));
		return factory;
	}

LGTM.

Perhaps some auto-configuration for micrometer-spring-legacy and Spring Boot 2 should be tracked in follow up tickets?

[matsumana]

@mweirauch Thank you for the confirmation.
I implemented JettyMetricsConfiguration.

[matsumana]

Would you merge this PR?
If this PR is merged, I'll create an issue about JettyMetricsConfiguration on Spring Boot's repo.

[mweirauch]

Just one thing, could you add JettyMetricsConfiguration to the metrics auto configuration so that it is picked up?

I am not too familiar with the EmbeddedServletContainerFactory setup within Spring Boot - one question pops up: Should the JettyEmbeddedServletContainerFactory actually be injected as well in case somebody rolls his own? (Spring Boot creates one if there is none.)

Anything else @checketts @jkschneider ? Rebase against 1.0.x?

[jkschneider]

Merged in 21a375a

[mweirauch]

I fear the JettyMetricsConfiguration::instrumentJettyThreadPool is kicking in too late. EmbeddedWebApplicationContext is calling the factory before afformentioned method is triggered.

Haven't found any other solution except actually defining the factory ourselfs. Dunno if the setup has changed with Spring Boot 2 and it becomes easier.

In case one really needs to roll his own factory and still wants threadpool instrumentation, one needs to exclude the JettyMetricsConfiguration from the auto-configuration. (e.g. with spring.autoconfigure.exclude)

[matsumana]

@mweirauch Thank you for the opinion.
I'll investigate.

[wilkinsona]

I'm looking at incorporating this change into Spring Boot 2.0. What's the benefit of InstrumentedQueuedThreadPool? I'm concerned about the need to set Jetty's thread pool as it will overwrite a user's customisation of the thread pool. It also limits the usefulness of JettyServerThreadPoolMetrids. Could it not take a QueuedThreadPool or even a ThreadPool? It could then adapt the meters that it binds based on the pool's concrete type.

[mweirauch]

So, perhaps we could actually ask for the ThreadPool in the current JettyMetricsPostProcessor.

Would it be allowed to register a new bean within postProcessAfterInitialization() on the held ApplicationContext? In this case a ThreadPool-adaptive variant of the JettyServerThreadPoolMetrics which can later be picked up by the MeterRegistry postprocessing where all MeterBinder are collected and bound.

Edit: Think we could just directly bind to the MeterRegistry.

[wilkinsona]

Would it be allowed to register a new bean within postProcessAfterInitialization() on the held ApplicationContext.

Not really, no. A BeanPostProcessor is intended for post-processing individual beans. A BeanFactoryPostProcessor would be a more appropriate place to perform bean registration. However, I'd prefer that this is done without any post-processing. It should be possible to do it in a manner similar to Boot's TomcatMetricsAutoConfiguration.

[matsumana]

How does this look like?
Although I've only tried Spring MVC yet, it seems to work fine

@Configuration
@ConditionalOnClass({ Server.class, Loader.class, WebAppContext.class })
public class JettyMetricsAutoConfiguration {

	@Bean
	public WebServerFactoryCustomizer<JettyServletWebServerFactory> jettyFactoryCustomizer(
			MeterRegistry registry) {
		return (factory) -> factory.setThreadPool(
				new InstrumentedQueuedThreadPool(registry, Collections.emptyList()));
	}
}

[wilkinsona]

That's better in terms of the conditions, but it's still setting the factory's thread pool and using InstrumentedQueueThreadPool. I think both of those should be avoided for the reasons I tried to explain above.

[wilkinsona]

I've opened #911 to describe and further discuss what I think needs to be done.