Prometheus Listener for Jmeter
This JMeter plugin is highly configurable listener (and config element) to allow users define they're own metrics (names, types etc.) and expose them through a Prometheus /metrics API to be scraped by a Prometheus server.
More documentation can be found on this project's wiki.
Here's a simple example to get us started. This example can be found here. All the documentation on this README is from this jmx file.
If we look closer at the very first Prometheus listener, it looks like the image below.
Let's go through all the columns one by one.
- Name: the name of the metric.
- Help: The help message of the metric.
- Labels: A comma seperated list of labels you want to apply to the metric.
labelis a keyword. In JMeter it means the name of the sampler.
codeis a keyword. It's the response code of the result.
- JMeter variables can be used here. See the section below.
- Type: The type of metric you're creating.
- Buckets of Quantiles:
- Buckets are comma seperated list of numbers. Can be integers or decimals.
- Quantiles are comma
,separated pair of decimals separated by a vertical bar
|. The first decimal being the quantile and the second being the error rating.
- Listen To: Dropdown to listen to samples or assertions. This only applies to Counters and SuccessRatio type metrics.
- Measuring: Dropdown menu of all the things you can measure
- See the Type and Measuring compatibility matrix section below.
Using JMeter variables as labels
Notice in the image above
jsr223_rt_as_summary (the 2nd down) has
category,label in it's labels column.
This plugin allows you to use variables in the test plan as label values for a given metric. You can see here in the above image, I simply generated a random string and assigned it to the
category jmeter variable, and this plugin exposed it as a label.
As you can see below, it produced that metric with the
category="[A,B,C]". Again, this example can be found here.
Success ratio is a concept specific to this plugin library. Often we want measure success rates of samplers and it's difficult to do so when the failure for a given metric or labelset has never occurred. It's difficult because it involves computations with NaNs.
So, we want to measure success ratios and be sure to emit zeros for both failures and success when the other is created for the first time. That way you can always safely run computations like rate() and so on.
Here you can see from the example
jsr223_can_fail turned into 3 metrics. The names have appeneded
_total. They're all counter type metrics that increment on success failure and total, respectively.
From this example you can see that
4 success + 2 failures = 6 total. Again, this metric guarantees a zero, for success or failure, for a metric that has a total of one or more.
Type and Measuring compatibility matrix
Does it make sense to have a Counter measuring Response Time? No. Does it make sense to have a Histogram of total successes? No.
This is a matrix of what metric types can measure what metrics. If you configure, say a histogram to measure count total, the plugin will likely do nothing to update that metric.
Bold types can listen to samples or assertions (not both at the same time). Note that if you don't use
label when listening to assertions you may get strange results. This is because one sample can generate many assertion results which are then counted. When there's no label to distinguish those counts, they'll be summed together which may or may not be expected.
What about gauges
I'm not quite sure how Guages make sense in the plugin. If you have a use case, I'd love to hear it. I wrote them in without actually having one, so you can technically create one, I just don't know how the listener may update it.
This library provides not only a listener, but a configuration element as well. This is useful when users have to make some computation for a specific use case and then want to expose that metric in Prometheus.
These use cases for this are mostly functional, where say you're validating something about a response that's specific the thing under test.
It works like this. Define a metric with this configuration element, and at test run-time this library will place that object in the jmeter variables from which you can access it.
Let's consider this use case. You, have an API that tells you the current number animals categorized by
mammal - perhaps running down your street. You may create a counter like this as you watch them go by.
So I then access the object I've created above like so. This is an absolutely trivial case of randomly generating the variables but the example is only trying to convey how one may interact with the objects created by the configuration element. However you extract the data, you can access and interact with the Prometheus metric you've created.
See the Prometheus Javadocs for more information on their API.
Which will expose a counter with all the appropriate labels.
Skipping samplers or other elements
You can re-use metrics in multiple listeners so long as they're defined in the exact same way. There will be undefined behavior if two or more listeners have the same metric (the same metric name) with different configurations.
Here you see
second_random_sampler in the labels of this metric, but you do not see
want_to_skip, the thing that we're trying to skip. Note the composition of the test plan as shown at the top of this page
This plugin has limited "out of the box" functionality because it gives you, the user, total control over what the metric names may be. That said, here's a sample dashboard given here in examples such that if you have a local prometheus/grafana stack you can a dashboard that looks something like this.
Properties you can override
|prometheus.port||9270||The port the http server will bind to|
|prometheus.delay||0||The delay (in seconds) the http server will wait before being destroyed|
|prometheus.save.threads||true||True or false value to save and collect jmeter thread metrics|
|prometheus.save.threads.name||jmeter_threads||The name of the metric describing jmeter threads|
|prometheus.save.jvm||true||Collect metrics from the JVM|
To build, simply maven package:
mvn clean package
This creates 2 jars, a shaded jar that has all the dependencies within it (this is the one you want) and the original jar. Both are in the target directory. Simply move the jar to your $JMETER_HOME/lib/ext directory as with any JMeter plugin and you're ready to go!
Feel free to open issues against this project, even to ask questions.