Skip to content

Commit

Permalink
Issue ReactiveX#354: Adapt prometheus metrics to the new format (Reac…
Browse files Browse the repository at this point in the history
…tiveX#375)

* Introduce new prometheus metrics collector for bulkheads

* Introduce new prometheus metrics collector for circuit breakers

* Introduce new prometheus metrics collector for rate limiters

* Avoid label list creation on every collect

* Adapt/extend prometheus documentation

* Improve metrics descriptions
  • Loading branch information
voievodin authored and RobWin committed Mar 28, 2019
1 parent e4cdf2c commit f951996
Show file tree
Hide file tree
Showing 11 changed files with 969 additions and 19 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -4,30 +4,43 @@

Integration with https://github.com/prometheus/client_java[Prometheus simple client]

Module provides exporters for `CircuitBreaker` and `RateLimiter` metrics.
Module provides exporters for `CircuitBreaker`, `RateLimiter` and `Bulkhead` metrics.
Every metric has `name` label indicating the relation to certain object, object category
such as `bulkhead` is inlined to the metric name, so that the combination of metric name + `name` label
uniquely identifies metric per backend. For example:

For the circuit breaker library exports 2 metrics:
```
resilience4j_bulkhead_available_concurrent_calls{name="Backend1"}
resilience4j_circuitbreaker_buffered_calls{name="Backend1"}
resilience4j_ratelimiter_waiting_threads{name=Backend1"}
```

1. By state with default metric name `circuit_breaker_states` and label `state`:
For every `CircuitBreaker` the library exports the following metrics:

- `closed`
- `open`
- `half_open`
1. `resilience4j_circuitbreaker_state` - the state of the circuit breaker, possible values:

2. By call result with default metric name `circuit_breaker_calls` and label `call_result`:
- 0 - CLOSED
- 1 - OPEN
- 2 - HALF_OPEN

- `successful`
- `failed`
- `not_permitted`
- `buffered`
- `buffered_max`
2. `resilience4j_circuitbreaker_calls` - the number of recorded calls,
the additional `kind` label indicates type of calls being recorded, the possible
`kind` values are `successful`, `failed` or `not_permitted`.

For the rate limiter following metric with default name `rate_limiter` and label `param` exported:
3. `resilience4j_circuitbreaker_buffered_calls` - the number of buffered calls

- `available_permissions`
- `waiting_threads`
4. `resilience4j_circuitbreaker_max_buffered_calls` - the maximum number of buffered calls

For every `RateLimiter` the library exports the following metrics:

1. `resilience4j_ratelimiter_available_permissions` - the number of available permissions
2. `resilience4j_ratelimiter_waiting_threads` - the number of waiting threads

For every `Bulkhead` the library exports the following metrics:

1. `resilience4j_bulkhead_available_concurrent_calls` - the number of available concurrent calls
2. `resilience4j_bulkhead_max_allowed_concurrent_calls` - the maximum number of allowed concurrent calls

The names of the rate limiters and circuit breakers are exposed using label `name`.

This module also provides `CallMeter` -- a composite metric to measure single call/request metrics such as:
- execution time distribution,
Expand Down Expand Up @@ -56,7 +69,7 @@ final CircuitBreaker foo = circuitBreakerRegistry.circuitBreaker("foo");
final CircuitBreaker boo = circuitBreakerRegistry.circuitBreaker("boo");

// Registering metrics in prometeus CollectorRegistry
collectorRegistry.register(CircuitBreakerExports.ofCircuitBreakerRegistry(circuitBreakerRegistry));
collectorRegistry.register(CircuitBreakerMetricsCollector.ofCircuitBreakerRegistry(circuitBreakerRegistry));
--

===== RateLimiter
Expand All @@ -71,10 +84,26 @@ final RateLimiter foo = rateLimiterRegistry.rateLimiter("foo");
final RateLimiter boo = rateLimiterRegistry.rateLimiter("boo");

// Registering metrics in prometeus CollectorRegistry
collectorRegistry.register(RateLimiterExports.ofRateLimiterRegistry(rateLimiterRegistry));
collectorRegistry.register(RateLimiterMetricsCollector.ofRateLimiterRegistry(rateLimiterRegistry));
--

===== Bulkhead

[source,java]
--
final CollectorRegistry collectorRegistry = CollectorRegistry.defaultRegistry;

final BulkheadRegistry bulkheadRegistry = BulkheadRegistry.ofDefaults();

final Bulkhead foo = bulkheadRegistry.bulkhead("foo");
final Bulkhead boo = bulkheadRegistry.bulkhead("boo");

// Registering metrics in prometeus CollectorRegistry
collectorRegistry.register(BulkheadMetricsCollector.ofBulkheadRegistry(rateLimiterRegistry));
--

For both it is possible to use just a collection of breakers and limiters instead of registry.
Every collector exposes more methods for binding corresponding objects, for example,
you can use suppliers, collections or even collect metrics for a single object.

===== Call Meter

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,10 @@
/**
* An adapter from builtin {@link Bulkhead.Metrics} to prometheus
* {@link io.prometheus.client.CollectorRegistry}.
*
* @deprecated use {@link io.github.resilience4j.prometheus.collectors.BulkheadMetricsCollector} instead
*/
@Deprecated
public class BulkheadExports extends Collector {
private static final String DEFAULT_NAME = "resilience4j_bulkhead";

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,10 @@
* {@link io.prometheus.client.CollectorRegistry}.
*
* Also exports {@link CircuitBreaker} state as a labeled metric
*
* @deprecated use {@link io.github.resilience4j.prometheus.collectors.CircuitBreakerMetricsCollector} instead.
*/
@Deprecated
public class CircuitBreakerExports extends Collector {

private static final String DEFAULT_NAME = "resilience4j_circuitbreaker";
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,10 @@
/**
* An adapter from builtin {@link RateLimiter.Metrics} to prometheus
* {@link io.prometheus.client.CollectorRegistry}.
*
* @deprecated use {@link io.github.resilience4j.prometheus.collectors.RateLimiterMetricsCollector} instead.
*/
@Deprecated
public class RateLimiterExports extends Collector {
private static final String DEFAULT_NAME = "resilience4j_ratelimiter";

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
/*
* Copyright 2019 Yevhenii Voievodin
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package io.github.resilience4j.prometheus.collectors;

import io.github.resilience4j.bulkhead.Bulkhead;
import io.github.resilience4j.bulkhead.Bulkhead.Metrics;
import io.github.resilience4j.bulkhead.BulkheadRegistry;
import io.prometheus.client.Collector;
import io.prometheus.client.GaugeMetricFamily;

import java.util.List;
import java.util.Objects;
import java.util.function.Supplier;

import static java.util.Arrays.asList;
import static java.util.Collections.singletonList;
import static java.util.Objects.requireNonNull;

/** Collects bulkhead exposed {@link Metrics}. */
public class BulkheadMetricsCollector extends Collector {

/**
* Creates a new collector with custom metric names and
* using given {@code supplier} as source of bulkheads.
*
* @param names the custom metric names
* @param supplier the supplier of bulkheads, note that supplier will be called one every {@link #collect()}
*/
public static BulkheadMetricsCollector ofSupplier(MetricNames names, Supplier<? extends Iterable<? extends Bulkhead>> supplier) {
return new BulkheadMetricsCollector(names, supplier);
}

/**
* Creates a new collector using given {@code supplier} as source of bulkheads.
*
* @param supplier the supplier of bulkheads, note that supplier will be called one very {@link #collect()}
*/
public static BulkheadMetricsCollector ofSupplier(Supplier<? extends Iterable<? extends Bulkhead>> supplier) {
return new BulkheadMetricsCollector(MetricNames.ofDefaults(), supplier);
}

/**
* Creates a new collector using given {@code registry} as source of bulkheads.
*
* @param registry the source of bulkheads
*/
public static BulkheadMetricsCollector ofBulkheadRegistry(BulkheadRegistry registry) {
return new BulkheadMetricsCollector(MetricNames.ofDefaults(), registry::getAllBulkheads);
}

/**
* Creates a new collector using given {@code bulkheads} iterable as source of bulkheads.
*
* @param bulkheads the source of bulkheads
*/
public static BulkheadMetricsCollector ofIterable(Iterable<? extends Bulkhead> bulkheads) {
return new BulkheadMetricsCollector(MetricNames.ofDefaults(), () -> bulkheads);
}

/**
* Creates a new collector for a given {@code bulkhead}.
*
* @param bulkhead the bulkhead to collect metrics for
*/
public static BulkheadMetricsCollector ofBulkhead(Bulkhead bulkhead) {
return ofIterable(singletonList(bulkhead));
}

private final MetricNames names;
private final Supplier<? extends Iterable<? extends Bulkhead>> bulkheadsSupplier;

private BulkheadMetricsCollector(MetricNames names, Supplier<? extends Iterable<? extends Bulkhead>> bulkheadsSupplier) {
this.names = Objects.requireNonNull(names);
this.bulkheadsSupplier = Objects.requireNonNull(bulkheadsSupplier);
}

@Override
public List<MetricFamilySamples> collect() {
GaugeMetricFamily availableCallsFamily = new GaugeMetricFamily(
names.getAvailableConcurrentCallsMetricName(),
"The number of available concurrent calls",
LabelNames.NAME
);
GaugeMetricFamily maxAllowedCallsFamily = new GaugeMetricFamily(
names.getMaxAllowedConcurrentCallsMetricName(),
"The maximum number of allowed concurrent calls",
LabelNames.NAME
);

for (Bulkhead bulkhead: bulkheadsSupplier.get()) {
List<String> labelValues = singletonList(bulkhead.getName());
availableCallsFamily.addMetric(labelValues, bulkhead.getMetrics().getAvailableConcurrentCalls());
maxAllowedCallsFamily.addMetric(labelValues, bulkhead.getMetrics().getMaxAllowedConcurrentCalls());
}

return asList(availableCallsFamily, maxAllowedCallsFamily);
}

/** Defines possible configuration for metric names. */
public static class MetricNames {

public static final String DEFAULT_BULKHEAD_AVAILABLE_CONCURRENT_CALLS_METRIC_NAME = "resilience4j_bulkhead_available_concurrent_calls";
public static final String DEFAULT_BULKHEAD_MAX_ALLOWED_CONCURRENT_CALLS_METRIC_NAME = "resilience4j_bulkhead_max_allowed_concurrent_calls";

/**
* Returns a builder for creating custom metric names.
* Note that names have default values, so only desired metrics can be renamed.
*/
public static Builder custom() {
return new Builder();
}

/** Returns default metric names. */
public static MetricNames ofDefaults() {
return new MetricNames();
}

private String availableConcurrentCallsMetricName = DEFAULT_BULKHEAD_AVAILABLE_CONCURRENT_CALLS_METRIC_NAME;
private String maxAllowedConcurrentCallsMetricName = DEFAULT_BULKHEAD_MAX_ALLOWED_CONCURRENT_CALLS_METRIC_NAME;

private MetricNames() {}

/**
* Returns the metric name for bulkhead concurrent calls,
* defaults to {@value DEFAULT_BULKHEAD_AVAILABLE_CONCURRENT_CALLS_METRIC_NAME}.
*/
public String getAvailableConcurrentCallsMetricName() {
return availableConcurrentCallsMetricName;
}

/**
* Returns the metric name for bulkhead max available concurrent calls,
* defaults to {@value DEFAULT_BULKHEAD_MAX_ALLOWED_CONCURRENT_CALLS_METRIC_NAME}.
*/
public String getMaxAllowedConcurrentCallsMetricName() {
return maxAllowedConcurrentCallsMetricName;
}

/** Helps building custom instance of {@link MetricNames}. */
public static class Builder {

private final MetricNames metricNames = new MetricNames();

/** Overrides the default metric name {@value MetricNames#DEFAULT_BULKHEAD_AVAILABLE_CONCURRENT_CALLS_METRIC_NAME} with a given one. */
public Builder availableConcurrentCallsMetricName(String availableConcurrentCallsMetricNames) {
metricNames.availableConcurrentCallsMetricName = requireNonNull(availableConcurrentCallsMetricNames);
return this;
}

/** Overrides the default metric name {@value MetricNames#DEFAULT_BULKHEAD_MAX_ALLOWED_CONCURRENT_CALLS_METRIC_NAME} with a given one. */
public Builder maxAllowedConcurrentCallsMetricName(String maxAllowedConcurrentCallsMetricName) {
metricNames.maxAllowedConcurrentCallsMetricName = requireNonNull(maxAllowedConcurrentCallsMetricName);
return this;
}

/** Builds {@link MetricNames} instance. */
public MetricNames build() {
return metricNames;
}
}
}
}
Loading

0 comments on commit f951996

Please sign in to comment.