New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add metrics for active connections on the client side #5288
Conversation
Motivation: There is no metrics for active connections on the client side out of the box. The number have to be computed by metric queries such as PromQL by substracting `closed` from `opened`. Otherwise, a custom metric should be implemented by users. Modifications: - Record `armeria.client.connections#value{state="active"}` in `MetricCollectingConnectionPoolListener` to get the active connections for a remote peer. Result: You can now monitor the number of active connections with `armeria.client.connections#value{state="active"}`
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @ikhoon 🙇 👍 🙇
final Meters meters = metersMap.get(commonTags); | ||
if (meters != null) { | ||
meters.decrement(); | ||
assert meters.activeConnections() >= 0 : "active connections should not be negative" + meters; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Question) Will the AssertionError
be caught anywhere?
assert meters.activeConnections() >= 0 : "active connections should not be negative" + meters; | |
checkState(meters.activeConnections() >= 0, : "active connections should not be negative" + meters); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The AssertionError
should be logged by:
armeria/core/src/main/java/com/linecorp/armeria/client/HttpChannelPool.java
Lines 516 to 523 in 9823b46
try { | |
listener.connectionClosed(protocol, remoteAddr, localAddr, channel); | |
} catch (Exception e) { | |
if (logger.isWarnEnabled()) { | |
logger.warn("{} Exception handling {}.connectionClosed()", | |
channel, listener.getClass().getName(), e); | |
} | |
} |
I believe
assert
would be proper because if the value becomes negative, it is a bug but not a bad state.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see, I thought Error
didn't inherit from Exception
. No problem if it is somehow logged and propagated to the user
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point. I don't see we need to log only Exception
s. Let me change it to Throwable
.
core/src/main/java/com/linecorp/armeria/client/ConnectionPoolMetrics.java
Show resolved
Hide resolved
core/src/main/java/com/linecorp/armeria/client/ConnectionPoolListener.java
Show resolved
Hide resolved
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #5288 +/- ##
===========================================
+ Coverage 0 73.95% +73.95%
- Complexity 0 20104 +20104
===========================================
Files 0 1730 +1730
Lines 0 74178 +74178
Branches 0 9464 +9464
===========================================
+ Hits 0 54855 +54855
- Misses 0 14849 +14849
- Partials 0 4474 +4474 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left small suggestions.
Thanks!
core/src/main/java/com/linecorp/armeria/client/ConnectionPoolMetrics.java
Outdated
Show resolved
Hide resolved
core/src/main/java/com/linecorp/armeria/client/ConnectionPoolMetrics.java
Show resolved
Hide resolved
core/src/main/java/com/linecorp/armeria/client/ConnectionPoolListener.java
Outdated
Show resolved
Hide resolved
I didn't think |
…etrics.java Co-authored-by: minux <songmw725@gmail.com>
…istener.java Co-authored-by: minux <songmw725@gmail.com>
Motivation: An `IllegalArgumentException` is raised when `ConnectionPoolListener.metricCollecting()` is used to monitor connection pools. The changes in line#5288 caused the problem. The different types (`Gauge`, `Counter`) of meter are registered with the same name. Unfortunately, it wasn't caught by unit tests because the restriction only exists in `PrometheusMeterRegistry`. `SimpleMeterRegistry` was used to verify the code. ``` java.lang.IllegalArgumentException: Failed to register Collector of type MicrometerCollector: armeria_client_connections is already in use by another Collector of type MicrometerCollector at io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:57) at io.prometheus.client.Collector.register(Collector.java:307) at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$applyToCollector$17(PrometheusMeterRegistry.java:557) at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1947) at io.micrometer.prometheus.PrometheusMeterRegistry.applyToCollector(PrometheusMeterRegistry.java:552) at io.micrometer.prometheus.PrometheusMeterRegistry.newGauge(PrometheusMeterRegistry.java:327) ... at io.micrometer.core.instrument.MeterRegistry.gauge(MeterRegistry.java:311) at io.micrometer.core.instrument.Gauge$Builder.register(Gauge.java:195) at com.linecorp.armeria.client.ConnectionPoolMetrics$Meters.<init>(ConnectionPoolMetrics.java:118) at com.linecorp.armeria.client.ConnectionPoolMetrics.lambda$increaseConnOpened$0(ConnectionPoolMetrics.java:61) ``` Modifications: - Infix `active` to the meter name and remove `status=active` tag. - `<prefix>.active.connections` now becomes the meter name for active connections. - `.connections` suffix is added to connection pool metrics. Result: - You no longer see `IllegalArgumentException` when using `ConnectionPoolListener.metricCollecting()`. - Breaking) `.connections` suffix is now automatically added to connection pool metrics. If you use a custom `MeterIdPrefix` with `ConnectionPoolListener.metricCollecting(MeterRegistry,MeterIdPrefix)`, the meter name will change.
Motivation: An `IllegalArgumentException` is raised when `ConnectionPoolListener.metricCollecting()` is used to monitor connection pools. The changes in #5288 caused the problem. The different types (`Gauge`, `Counter`) of meters are registered with the same name. Unfortunately, it wasn't caught by unit tests because the restriction only exists in `PrometheusMeterRegistry`. `SimpleMeterRegistry` was used to verify the code. ```java java.lang.IllegalArgumentException: Failed to register Collector of type MicrometerCollector: armeria_client_connections is already in use by another Collector of type MicrometerCollector at io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:57) at io.prometheus.client.Collector.register(Collector.java:307) at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$applyToCollector$17(PrometheusMeterRegistry.java:557) at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1947) at io.micrometer.prometheus.PrometheusMeterRegistry.applyToCollector(PrometheusMeterRegistry.java:552) at io.micrometer.prometheus.PrometheusMeterRegistry.newGauge(PrometheusMeterRegistry.java:327) ... at io.micrometer.core.instrument.MeterRegistry.gauge(MeterRegistry.java:311) at io.micrometer.core.instrument.Gauge$Builder.register(Gauge.java:195) at com.linecorp.armeria.client.ConnectionPoolMetrics$Meters.<init>(ConnectionPoolMetrics.java:118) at com.linecorp.armeria.client.ConnectionPoolMetrics.lambda$increaseConnOpened$0(ConnectionPoolMetrics.java:61) ``` Modifications: - Infix `active` to the meter name and remove `status=active` tag. - `<prefix>.active.connections` now becomes the meter name for active connections. - `.connections` suffix is added to connection pool metrics. Result: - You no longer see `IllegalArgumentException` when using `ConnectionPoolListener.metricCollecting()`. - Breaking) `.connections` suffix is now automatically added to connection pool metrics. If you use a custom `MeterIdPrefix` with `ConnectionPoolListener.metricCollecting(MeterRegistry,MeterIdPrefix)`, the meter name will change.
…#5466) Motivation: An `IllegalArgumentException` is raised when `ConnectionPoolListener.metricCollecting()` is used to monitor connection pools. The changes in line#5288 caused the problem. The different types (`Gauge`, `Counter`) of meters are registered with the same name. Unfortunately, it wasn't caught by unit tests because the restriction only exists in `PrometheusMeterRegistry`. `SimpleMeterRegistry` was used to verify the code. ```java java.lang.IllegalArgumentException: Failed to register Collector of type MicrometerCollector: armeria_client_connections is already in use by another Collector of type MicrometerCollector at io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:57) at io.prometheus.client.Collector.register(Collector.java:307) at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$applyToCollector$17(PrometheusMeterRegistry.java:557) at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1947) at io.micrometer.prometheus.PrometheusMeterRegistry.applyToCollector(PrometheusMeterRegistry.java:552) at io.micrometer.prometheus.PrometheusMeterRegistry.newGauge(PrometheusMeterRegistry.java:327) ... at io.micrometer.core.instrument.MeterRegistry.gauge(MeterRegistry.java:311) at io.micrometer.core.instrument.Gauge$Builder.register(Gauge.java:195) at com.linecorp.armeria.client.ConnectionPoolMetrics$Meters.<init>(ConnectionPoolMetrics.java:118) at com.linecorp.armeria.client.ConnectionPoolMetrics.lambda$increaseConnOpened$0(ConnectionPoolMetrics.java:61) ``` Modifications: - Infix `active` to the meter name and remove `status=active` tag. - `<prefix>.active.connections` now becomes the meter name for active connections. - `.connections` suffix is added to connection pool metrics. Result: - You no longer see `IllegalArgumentException` when using `ConnectionPoolListener.metricCollecting()`. - Breaking) `.connections` suffix is now automatically added to connection pool metrics. If you use a custom `MeterIdPrefix` with `ConnectionPoolListener.metricCollecting(MeterRegistry,MeterIdPrefix)`, the meter name will change.
Motivation:
There are no metrics for active connections on the client side out of the box. The number has to be computed by metric queries such as PromQL by subtracting
closed
fromopened
.Otherwise, a custom metric should be implemented by users.
Modifications:
armeria.client.connections#value{state="active"}
inMetricCollectingConnectionPoolListener
to get the active connections for a remote peer.Result:
armeria.client.connections#value{state="active"}