File descriptors opened by jmx exporter getting stuck in CLOSE_WAIT state

[anilsomisetty]

I am using trino where I am using the jmx exporter of version 19 to expose my application metrics over a http port for my prometheus agent to collect these metrics.

I have my prometheus agent configured to collect metrics from the exposed http port with a specific timeout.

My issue: File descriptors are stuck in CLOSE_WAIT state and their count keeps on increasing, after the file descriptor count reaches a specific limit my application is crashing and becoming unresponsive.

In my case what's happening is the file descriptor connections that has been opened between jmx exporter and my prometheus metric collection agent are getting stuck in CLOSE_WAIT state because my agent is exiting after the timeout and the exporter is not able to provide the metrics within specified timeout.

The exporter is stuck in writing metrics to the descriptor because on the other side agent reading the metrics has left and ByteArrayOutputStream is stuck in write call and the file descriptor is not getting closed at this line of code

https://github.com/prometheus/client_java/blob/ed0d7ae3b57a3986f6531d1a37db031a331227e6/simpleclient_httpserver/src/main/java/io/prometheus/client/exporter/HTTPServer.java#L126

My agent collecting the metrics leaves after specified timeout, but ideally the exporter should understand that the requestor has left and should close the connection and shouldn't get stuck in write call and shouldn't keep the fd open.

I understand that once the agent leaves after timeout and exporter will not be able to give it the entire metrics of that moment and the metrics will be lost at that timeframe which is fine for me.

Please help me with this.

[rwilliams-r7]

We are also having this problem. We have found it can be so bad it will stop the application running in this case a cluster of Trino instances. Do you have any time to work on this? We had to remove it from our production env.

[dhoard]

The core HTTP server code is in the Prometheus client_java library. It will have to expose configuration to allow setting the socket timeout. I still have concerns around the RMI socket timeouts.

prometheus-metrics-exporter-httpserver

Relevant class: https://github.com/prometheus/client_java/blob/main/prometheus-metrics-exporter-httpserver/src/main/java/io/prometheus/metrics/exporter/httpserver/HTTPServer.java

I create an issue for the client_java library (prometheus/client_java#924) Once added, the configuration can be exposed via the exporter configuration.

[anilsomisetty]

I have tested my application with the latest changes available in main branch of the exporter which uses latest version of prometheus http server from client java repository and I see that when I'm using that my file descriptor connections stuck in CLOSE_WAIT state are getting cleared, could you please confirm what changes have been made that the fd's are getting cleared without adding the timeout ?
When would be the official release of the latest version of jmx exporter which is integrated with client_java version of v1.1.0 ?

[dhoard]

@anilsomisetty I am not aware of any direct changes in client_java v1.1.0 that would have changed the behavior regarding CLOSE_WAIT sockets.

We are targeting a beta release (due to the major changes made in client_java) of the main code in the next 2 weeks.

[anilsomisetty]

@dhoard I see that due to the changes that have been made in this commit of client java code prometheus/client_java@1966186 after version 0.16.0 release the connections getting stuck in CLOSE_WAIT state are getting cleared after 2 minutes as they are in idle state.

I have tested the 0.19.0 version of jmx exporter integrated with 0.16.0 release of client java with this commit changes included and have seen that the close wait stuck connection are getting closed which was not the case without this commit changes.

[dhoard]

@anilsomisetty Interesting observation.

The issue with the code in prometheus/client_java@1966186 is that the ThreadPoolExecutor is unbounded.

The code in the latest release should behave the same way.

Is it possible for you build and test the main branch (unreleased) to see if you can reproduce the issue?

[anilsomisetty]

@dhoard I have build the main branch code of jmx exporter which is using client java version v1.1.0 and I'm not able to reproduce the issue, the stuck close wait connections are getting closed and are cleared.

In the latest code i see ThreadPoolExecutor has been bounded by corepoolsize and maxpoolsize so it makes this bounded am I right ?

[dhoard]

@anilsomisetty correct - the ThreadPoolExecutor is now bounded.

Thanks for testing the code in main (unreleased)!

Do you experience the issue with the latest release 0.20.0?

[anilsomisetty]

@dhoard yes, the issue persists in the latest release 0.20.0
As there is a default limit of max 10 threads set in this latest release when all 10 connections are getting stuck in close wait state no new connections are getting created.

Could you please tell me when is the plan to have a generally available official release if beta is planned in next 2 weeks ?