Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add metrics for active connections on the client side #5288

Merged
merged 7 commits into from Jan 24, 2024

Conversation

ikhoon
Copy link
Contributor

@ikhoon ikhoon commented Nov 6, 2023

Motivation:

There are no metrics for active connections on the client side out of the box. The number has to be computed by metric queries such as PromQL by subtracting closed from opened.
Otherwise, a custom metric should be implemented by users.

Modifications:

  • Record armeria.client.connections#value{state="active"} in MetricCollectingConnectionPoolListener to get the active connections for a remote peer.

Result:

Motivation:

There is no metrics for active connections on the client side out of the
box. The number have to be computed by metric queries such as PromQL by
substracting `closed` from `opened`.
Otherwise, a custom metric should be implemented by users.

Modifications:

- Record `armeria.client.connections#value{state="active"}` in
  `MetricCollectingConnectionPoolListener` to get the active connections
  for a remote peer.

Result:

You can now monitor the number of active connections with
`armeria.client.connections#value{state="active"}`
@ikhoon ikhoon added this to the 1.27.0 milestone Nov 6, 2023
Copy link
Contributor

@jrhee17 jrhee17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ikhoon 🙇 👍 🙇

final Meters meters = metersMap.get(commonTags);
if (meters != null) {
meters.decrement();
assert meters.activeConnections() >= 0 : "active connections should not be negative" + meters;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question) Will the AssertionError be caught anywhere?

Suggested change
assert meters.activeConnections() >= 0 : "active connections should not be negative" + meters;
checkState(meters.activeConnections() >= 0, : "active connections should not be negative" + meters);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The AssertionError should be logged by:

try {
listener.connectionClosed(protocol, remoteAddr, localAddr, channel);
} catch (Exception e) {
if (logger.isWarnEnabled()) {
logger.warn("{} Exception handling {}.connectionClosed()",
channel, listener.getClass().getName(), e);
}
}

I believe assert would be proper because if the value becomes negative, it is a bug but not a bad state.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I thought Error didn't inherit from Exception. No problem if it is somehow logged and propagated to the user

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I don't see we need to log only Exceptions. Let me change it to Throwable.

Copy link

codecov bot commented Nov 7, 2023

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (fd31c66) 0.00% compared to head (c2588b7) 73.95%.
Report is 14 commits behind head on main.

Files Patch % Lines
...linecorp/armeria/client/ConnectionPoolMetrics.java 96.00% 0 Missing and 2 partials ⚠️
...a/com/linecorp/armeria/client/HttpChannelPool.java 0.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #5288       +/-   ##
===========================================
+ Coverage        0   73.95%   +73.95%     
- Complexity      0    20104    +20104     
===========================================
  Files           0     1730     +1730     
  Lines           0    74178    +74178     
  Branches        0     9464     +9464     
===========================================
+ Hits            0    54855    +54855     
- Misses          0    14849    +14849     
- Partials        0     4474     +4474     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@minwoox minwoox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left small suggestions.
Thanks!

@ikhoon
Copy link
Contributor Author

ikhoon commented Nov 30, 2023

Then, probably better to use Caffeine? If so, we could also limit the size or use weak key and value.

I didn't think Caffeine was a good idea because that may produce incorrect metric information. Weak key seems worth a try.

ikhoon and others added 2 commits November 30, 2023 19:48
…etrics.java

Co-authored-by: minux <songmw725@gmail.com>
…istener.java

Co-authored-by: minux <songmw725@gmail.com>
@ikhoon ikhoon merged commit ecdac50 into line:main Jan 24, 2024
15 checks passed
@ikhoon ikhoon deleted the active-client-connections-metrics branch January 24, 2024 08:39
ikhoon added a commit to ikhoon/armeria that referenced this pull request Feb 15, 2024
Motivation:

An `IllegalArgumentException` is raised when `ConnectionPoolListener.metricCollecting()`
is used to monitor connection pools. The changes in line#5288 caused the
problem.  The different types (`Gauge`, `Counter`) of meter are
registered with the same name. Unfortunately, it wasn't caught by unit
tests because the restriction only exists in `PrometheusMeterRegistry`.
`SimpleMeterRegistry` was used to verify the code.
```
java.lang.IllegalArgumentException: Failed to register Collector of type MicrometerCollector: armeria_client_connections is already in use by another Collector of type MicrometerCollector
	at io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:57)
	at io.prometheus.client.Collector.register(Collector.java:307)
	at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$applyToCollector$17(PrometheusMeterRegistry.java:557)
	at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1947)
	at io.micrometer.prometheus.PrometheusMeterRegistry.applyToCollector(PrometheusMeterRegistry.java:552)
	at io.micrometer.prometheus.PrometheusMeterRegistry.newGauge(PrometheusMeterRegistry.java:327)
        ...
	at io.micrometer.core.instrument.MeterRegistry.gauge(MeterRegistry.java:311)
	at io.micrometer.core.instrument.Gauge$Builder.register(Gauge.java:195)
	at com.linecorp.armeria.client.ConnectionPoolMetrics$Meters.<init>(ConnectionPoolMetrics.java:118)
	at com.linecorp.armeria.client.ConnectionPoolMetrics.lambda$increaseConnOpened$0(ConnectionPoolMetrics.java:61)
```

Modifications:

- Infix `active` to the meter name and remove `status=active` tag.
  - `<prefix>.active.connections` now becomes the meter name for active
    connections.
- `.connections` suffix is added to connection pool metrics.

Result:

- You no longer see `IllegalArgumentException` when using
  `ConnectionPoolListener.metricCollecting()`.
- Breaking) `.connections` suffix is now automatically added to
  connection pool metrics. If you use a custom `MeterIdPrefix` with
  `ConnectionPoolListener.metricCollecting(MeterRegistry,MeterIdPrefix)`,
  the meter name will change.
jrhee17 pushed a commit that referenced this pull request Feb 21, 2024
Motivation:

An `IllegalArgumentException` is raised when
`ConnectionPoolListener.metricCollecting()` is used to monitor
connection pools. The changes in #5288 caused the problem. The different
types (`Gauge`, `Counter`) of meters are registered with the same name.
Unfortunately, it wasn't caught by unit tests because the restriction
only exists in `PrometheusMeterRegistry`. `SimpleMeterRegistry` was used
to verify the code.
```java
java.lang.IllegalArgumentException: Failed to register Collector of type MicrometerCollector: armeria_client_connections is already in use by another Collector of type MicrometerCollector
	at io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:57)
	at io.prometheus.client.Collector.register(Collector.java:307)
	at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$applyToCollector$17(PrometheusMeterRegistry.java:557)
	at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1947)
	at io.micrometer.prometheus.PrometheusMeterRegistry.applyToCollector(PrometheusMeterRegistry.java:552)
	at io.micrometer.prometheus.PrometheusMeterRegistry.newGauge(PrometheusMeterRegistry.java:327)
        ...
	at io.micrometer.core.instrument.MeterRegistry.gauge(MeterRegistry.java:311)
	at io.micrometer.core.instrument.Gauge$Builder.register(Gauge.java:195)
	at com.linecorp.armeria.client.ConnectionPoolMetrics$Meters.<init>(ConnectionPoolMetrics.java:118)
	at com.linecorp.armeria.client.ConnectionPoolMetrics.lambda$increaseConnOpened$0(ConnectionPoolMetrics.java:61)
```

Modifications:

- Infix `active` to the meter name and remove `status=active` tag.
- `<prefix>.active.connections` now becomes the meter name for active
connections.
- `.connections` suffix is added to connection pool metrics.

Result:

- You no longer see `IllegalArgumentException` when using
`ConnectionPoolListener.metricCollecting()`.
- Breaking) `.connections` suffix is now automatically added to
connection pool metrics. If you use a custom `MeterIdPrefix` with
`ConnectionPoolListener.metricCollecting(MeterRegistry,MeterIdPrefix)`,
the meter name will change.
jrhee17 pushed a commit to jrhee17/armeria that referenced this pull request Mar 4, 2024
…#5466)

Motivation:

An `IllegalArgumentException` is raised when
`ConnectionPoolListener.metricCollecting()` is used to monitor
connection pools. The changes in line#5288 caused the problem. The different
types (`Gauge`, `Counter`) of meters are registered with the same name.
Unfortunately, it wasn't caught by unit tests because the restriction
only exists in `PrometheusMeterRegistry`. `SimpleMeterRegistry` was used
to verify the code.
```java
java.lang.IllegalArgumentException: Failed to register Collector of type MicrometerCollector: armeria_client_connections is already in use by another Collector of type MicrometerCollector
	at io.prometheus.client.CollectorRegistry.register(CollectorRegistry.java:57)
	at io.prometheus.client.Collector.register(Collector.java:307)
	at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$applyToCollector$17(PrometheusMeterRegistry.java:557)
	at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1947)
	at io.micrometer.prometheus.PrometheusMeterRegistry.applyToCollector(PrometheusMeterRegistry.java:552)
	at io.micrometer.prometheus.PrometheusMeterRegistry.newGauge(PrometheusMeterRegistry.java:327)
        ...
	at io.micrometer.core.instrument.MeterRegistry.gauge(MeterRegistry.java:311)
	at io.micrometer.core.instrument.Gauge$Builder.register(Gauge.java:195)
	at com.linecorp.armeria.client.ConnectionPoolMetrics$Meters.<init>(ConnectionPoolMetrics.java:118)
	at com.linecorp.armeria.client.ConnectionPoolMetrics.lambda$increaseConnOpened$0(ConnectionPoolMetrics.java:61)
```

Modifications:

- Infix `active` to the meter name and remove `status=active` tag.
- `<prefix>.active.connections` now becomes the meter name for active
connections.
- `.connections` suffix is added to connection pool metrics.

Result:

- You no longer see `IllegalArgumentException` when using
`ConnectionPoolListener.metricCollecting()`.
- Breaking) `.connections` suffix is now automatically added to
connection pool metrics. If you use a custom `MeterIdPrefix` with
`ConnectionPoolListener.metricCollecting(MeterRegistry,MeterIdPrefix)`,
the meter name will change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add the gauge meter for the number of active client connections
3 participants