Skip to content

Commit

Permalink
Add histogram to simpleclient.
Browse files Browse the repository at this point in the history
  • Loading branch information
brian-brazil committed Feb 17, 2015
1 parent 4ce980f commit ed184d8
Show file tree
Hide file tree
Showing 12 changed files with 582 additions and 61 deletions.
55 changes: 33 additions & 22 deletions benchmark/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,14 @@ The main outcomes of the benchmarks:
* The original client is much slower than the Simpleclient or Codahale, especially when used concurrently.
* Codahale Meters are slower than Codahale/Simpleclient Counters
* Codahale and original client Summaries are 10x slower than other metrics.
* Simpleclient Histograms are 10-100X faster than Codahale and original client Summaries.
* Simpleclient `Gauge.Child.set` is relatively slow, especially when done concurrently.
* Label lookups in both Prometheus clients is relatively slow.
* Label lookups in both Prometheus clients are relatively slow.

Accordingly, in terms of client instrumentation performance I suggest the following:
* It's cheap to extensively instrument your code with Simpleclient Counters/Gauges/Summaries without labels, or Codahale Counters.
* Avoid Codahale Meters, in favour of Codahale/Simpleclient Counters and calculating the rate in your monitoring system (e.g. the `rate()` function in Prometheus).
* Avoid original client Summaries and Codahale Histograms/Timers on high update rate code paths.
* Use Simpleclient Histograms rather than original client Summaries and Codahale Histograms/Timers.
* Avoid the original client.
* For high update rate (>1000 per second) prometheus metrics using labels, you should cache the Child. Java 8 may make this better due to an improved ConcurrentHashMap implementation.
* If a use case appears for high update rate use of SimpleClient's `Gauge.Child.set`, we should alter `DoubleAdder` to more efficiently handle this use case.
Expand Down Expand Up @@ -116,33 +117,43 @@ Codahale lacks a metric with a `set` method, so we'll compare to `Counter` which

### Summaries

The simpleclient `Summary` doesn't have percentiles, so it's not an apples to
apples comparison. The closest to the original client's `Summary` is Codahale's
The simpleclient `Summary` doesn't have percentiles, simpleclient's `Histogram`
offers a way to calculate percentiles on the server side that works with aggregation.
The closest to the original client's `Summary` is Codahale's
`Timer`, but that includes timing calls so we compare with `Histogram` instead.

java -jar target/benchmarks.jar SummaryBenchmark -wi 5 -i 5 -f 1 -t 1
i.p.b.SummaryBenchmark.codahaleHistogramBenchmark avgt 5 187.192 ± 12.304 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryBenchmark avgt 5 61.140 ± 1.298 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryChildBenchmark avgt 5 15.322 ± 0.266 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryNoLabelsBenchmark avgt 5 15.750 ± 0.476 ns/op
i.p.b.SummaryBenchmark.prometheusSummaryBenchmark avgt 5 988.597 ± 343.221 ns/op
i.p.b.SummaryBenchmark.prometheusSummaryChildBenchmark avgt 5 1165.270 ± 916.324 ns/op
i.p.b.SummaryBenchmark.codahaleHistogramBenchmark avgt 5 186.306 ± 4.958 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleHistogramBenchmark avgt 5 81.595 ± 4.491 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleHistogramChildBenchmark avgt 5 22.143 ± 1.713 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleHistogramNoLabelsBenchmark avgt 5 22.066 ± 0.812 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryBenchmark avgt 5 59.588 ± 2.087 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryChildBenchmark avgt 5 15.300 ± 0.659 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryNoLabelsBenchmark avgt 5 15.608 ± 0.271 ns/op
i.p.b.SummaryBenchmark.prometheusSummaryBenchmark avgt 5 981.640 ± 315.146 ns/op
i.p.b.SummaryBenchmark.prometheusSummaryChildBenchmark avgt 5 1155.179 ± 850.237 ns/op

java -jar target/benchmarks.jar SummaryBenchmark -wi 5 -i 5 -f 1 -t 2
i.p.b.SummaryBenchmark.codahaleHistogramBenchmark avgt 5 243.749 ± 37.080 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryBenchmark avgt 5 119.753 ± 15.056 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryChildBenchmark avgt 5 32.614 ± 15.976 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryNoLabelsBenchmark avgt 5 32.627 ± 8.943 ns/op
i.p.b.SummaryBenchmark.prometheusSummaryBenchmark avgt 5 2021.984 ± 561.545 ns/op
i.p.b.SummaryBenchmark.prometheusSummaryChildBenchmark avgt 5 2338.371 ± 1515.886 ns/op
i.p.b.SummaryBenchmark.codahaleHistogramBenchmark avgt 5 289.245 ± 39.721 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleHistogramBenchmark avgt 5 127.014 ± 19.285 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleHistogramChildBenchmark avgt 5 52.597 ± 10.781 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleHistogramNoLabelsBenchmark avgt 5 53.295 ± 9.891 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryBenchmark avgt 5 117.810 ± 11.694 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryChildBenchmark avgt 5 31.933 ± 3.439 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryNoLabelsBenchmark avgt 5 33.918 ± 5.571 ns/op
i.p.b.SummaryBenchmark.prometheusSummaryBenchmark avgt 5 2059.498 ± 616.954 ns/op
i.p.b.SummaryBenchmark.prometheusSummaryChildBenchmark avgt 5 2346.163 ± 1503.034 ns/op

java -jar target/benchmarks.jar SummaryBenchmark -wi 5 -i 5 -f 1 -t 4
i.p.b.SummaryBenchmark.codahaleHistogramBenchmark avgt 5 559.505 ± 5.169 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryBenchmark avgt 5 137.072 ± 12.044 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryChildBenchmark avgt 5 44.228 ± 0.697 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryNoLabelsBenchmark avgt 5 41.223 ± 1.978 ns/op
i.p.b.SummaryBenchmark.prometheusSummaryBenchmark avgt 5 4023.354 ± 1122.317 ns/op
i.p.b.SummaryBenchmark.prometheusSummaryChildBenchmark avgt 5 4606.571 ± 3392.565 ns/op
i.p.b.SummaryBenchmark.codahaleHistogramBenchmark avgt 5 587.956 ± 2.788 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleHistogramBenchmark avgt 5 163.313 ± 5.163 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleHistogramChildBenchmark avgt 5 66.957 ± 1.746 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleHistogramNoLabelsBenchmark avgt 5 67.064 ± 1.681 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryBenchmark avgt 5 140.166 ± 4.263 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryChildBenchmark avgt 5 40.065 ± 0.138 ns/op
i.p.b.SummaryBenchmark.prometheusSimpleSummaryNoLabelsBenchmark avgt 5 41.331 ± 1.899 ns/op
i.p.b.SummaryBenchmark.prometheusSummaryBenchmark avgt 5 3950.152 ± 1214.866 ns/op
i.p.b.SummaryBenchmark.prometheusSummaryChildBenchmark avgt 5 4676.946 ± 3625.977 ns/op

Note the high error bars for the original client, it got slower with each iteration
so I suspect a flaw in the test setup.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@ public class SummaryBenchmark {
io.prometheus.client.Summary prometheusSimpleSummary;
io.prometheus.client.Summary.Child prometheusSimpleSummaryChild;
io.prometheus.client.Summary prometheusSimpleSummaryNoLabels;
io.prometheus.client.Histogram prometheusSimpleHistogram;
io.prometheus.client.Histogram.Child prometheusSimpleHistogramChild;
io.prometheus.client.Histogram prometheusSimpleHistogramNoLabels;

@Setup
public void setup() {
Expand All @@ -46,6 +49,17 @@ public void setup() {
.help("some description..")
.create();

prometheusSimpleHistogram = io.prometheus.client.Histogram.build()
.name("name")
.help("some description..")
.labelNames("some", "group").create();
prometheusSimpleHistogramChild = prometheusSimpleHistogram.labels("test", "group");

prometheusSimpleHistogramNoLabels = io.prometheus.client.Histogram.build()
.name("name")
.help("some description..")
.create();

registry = new MetricRegistry();
codahaleHistogram = registry.histogram("name");
}
Expand All @@ -70,7 +84,7 @@ public void prometheusSummaryChildBenchmark() {
public void prometheusSimpleSummaryBenchmark() {
prometheusSimpleSummary.labels("test", "group").observe(1) ;
}

@Benchmark
@BenchmarkMode({Mode.AverageTime})
@OutputTimeUnit(TimeUnit.NANOSECONDS)
Expand All @@ -85,6 +99,27 @@ public void prometheusSimpleSummaryNoLabelsBenchmark() {
prometheusSimpleSummaryNoLabels.observe(1);
}

@Benchmark
@BenchmarkMode({Mode.AverageTime})
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public void prometheusSimpleHistogramBenchmark() {
prometheusSimpleHistogram.labels("test", "group").observe(1) ;
}

@Benchmark
@BenchmarkMode({Mode.AverageTime})
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public void prometheusSimpleHistogramChildBenchmark() {
prometheusSimpleHistogramChild.observe(1);
}

@Benchmark
@BenchmarkMode({Mode.AverageTime})
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public void prometheusSimpleHistogramNoLabelsBenchmark() {
prometheusSimpleHistogramNoLabels.observe(1);
}

@Benchmark
@BenchmarkMode({Mode.AverageTime})
@OutputTimeUnit(TimeUnit.NANOSECONDS)
Expand Down
16 changes: 15 additions & 1 deletion simpleclient/src/main/java/io/prometheus/client/Collector.java
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,8 @@ public abstract class Collector {
public static enum Type {
COUNTER,
GAUGE,
SUMMARY
SUMMARY,
HISTOGRAM,
}

/**
Expand Down Expand Up @@ -163,4 +164,17 @@ protected static void checkMetricLabelName(String name) {
throw new IllegalArgumentException("Invalid metric label name, reserved for internal use: " + name);
}
}

/**
* Convert a double to it's string representation in Go.
*/
public static String doubleToGoString(double d) {
if (d == Double.POSITIVE_INFINITY) {
return "+Inf";
}
if (d == Double.NEGATIVE_INFINITY) {
return "-Inf";
}
return Double.toString(d);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ public class Counter extends SimpleCollector<Counter.Child, Counter> {
super(b);
}

public static class Builder extends SimpleCollector.Builder<Counter> {
public static class Builder extends SimpleCollector.Builder<Builder> {
@Override
public Counter create() {
return new Counter(this);
Expand Down
2 changes: 1 addition & 1 deletion simpleclient/src/main/java/io/prometheus/client/Gauge.java
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ public class Gauge extends SimpleCollector<Gauge.Child, Gauge> {
super(b);
}

public static class Builder extends SimpleCollector.Builder<Gauge> {
public static class Builder extends SimpleCollector.Builder<Builder> {
@Override
public Gauge create() {
return new Gauge(this);
Expand Down
Loading

0 comments on commit ed184d8

Please sign in to comment.