Skip to content
A Prometheus Listener for Apache JMeter that exposes results in an http API
Branch: master
Clone or download
Latest commit e0885c7 Mar 17, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docs #51 functionality for assertions Mar 17, 2019
licenses add prometheus license Feb 6, 2019
src
.gitignore version 0.4.0-SNAPSHOT begin Mar 15, 2019
.travis.yml add travis cl Jan 21, 2018
CHANGELOG.md #51 functionality for assertions Mar 17, 2019
CODE_OF_CONDUCT.md
CONTRIBUTING.md Create CONTRIBUTING.md Jul 20, 2017
LICENSE licensing info Mar 24, 2017
NOTICE add prometheus license Feb 6, 2019
README.md #51 functionality for assertions Mar 17, 2019
pom.xml next version Mar 18, 2019

README.md

Prometheus Listener for Jmeter

Overview

This JMeter plugin is highly configurable listener (and config element) to allow users define they're own metrics (names, types etc.) and expose them through a Prometheus /metrics API to be scraped by a Prometheus server.

Documentation

More documentation can be found on this project's wiki.

Listener QuickDoc

Here's a simple example to get us started. This example can be found here. All the documentation on this README is from this jmx file.

JMeter testplan

If we look closer at the very first Prometheus listener, it looks like the image below.

JMeter testplan

Let's go through all the columns one by one.

  • Name: the name of the metric.
  • Help: The help message of the metric.
  • Labels: A comma seperated list of labels you want to apply to the metric.
    • label is a keyword. In JMeter it means the name of the sampler.
    • code is a keyword. It's the response code of the result.
    • JMeter variables can be used here. See the section below.
  • Type: The type of metric you're creating.
  • Buckets of Quantiles:
    • Buckets are comma seperated list of numbers. Can be integers or decimals.
    • Quantiles are comma , separated pair of decimals separated by a vertical bar |. The first decimal being the quantile and the second being the error rating.
  • Listen To: Dropdown to listen to samples or assertions. This only applies to Counters and SuccessRatio type metrics.
  • Measuring: Dropdown menu of all the things you can measure

Using JMeter variables as labels

Notice in the image above jsr223_rt_as_summary (the 2nd down) has category,label in it's labels column.

This plugin allows you to use variables in the test plan as label values for a given metric. You can see here in the above image, I simply generated a random string and assigned it to the category jmeter variable, and this plugin exposed it as a label.

As you can see below, it produced that metric with the category="[A,B,C]". Again, this example can be found here.

JMeter testplan

Listener output

JMeter testplan

Success Ratio

Success ratio is a concept specific to this plugin library. Often we want measure success rates of samplers and it's difficult to do so when the failure for a given metric or labelset has never occurred. It's difficult because it involves computations with NaNs.

So, we want to measure success ratios and be sure to emit zeros for both failures and success when the other is created for the first time. That way you can always safely run computations like rate() and so on.

Here's prometheus documentation on why it's important. Here is this repositories issue for this feature.

Here you can see from the example jsr223_can_fail turned into 3 metrics. The names have appeneded _success, _failure and _total. They're all counter type metrics that increment on success failure and total, respectively.

From this example you can see that 4 success + 2 failures = 6 total. Again, this metric guarantees a zero, for success or failure, for a metric that has a total of one or more.

JMeter testplan

Type and Measuring compatibility matrix

Does it make sense to have a Counter measuring Response Time? No. Does it make sense to have a Histogram of total successes? No.

This is a matrix of what metric types can measure what metrics. If you configure, say a histogram to measure count total, the plugin will likely do nothing to update that metric.

Bold types can listen to samples or assertions (not both at the same time). Note that if you don't use label when listening to assertions you may get strange results. This is because one sample can generate many assertion results which are then counted. When there's no label to distinguish those counts, they'll be summed together which may or may not be expected.

Histogram Summary Counter Guage Success Ratio
Response time x x
Response size x x
Latency x x
Idle time x x
Connect time x x
Count total x
Failure total x
Success total x
Success Ratio x

What about gauges

I'm not quite sure how Guages make sense in the plugin. If you have a use case, I'd love to hear it. I wrote them in without actually having one, so you can technically create one, I just don't know how the listener may update it.

Config QuickDoc

This library provides not only a listener, but a configuration element as well. This is useful when users have to make some computation for a specific use case and then want to expose that metric in Prometheus.

These use cases for this are mostly functional, where say you're validating something about a response that's specific the thing under test.

It works like this. Define a metric with this configuration element, and at test run-time this library will place that object in the jmeter variables from which you can access it.

Let's consider this use case. You, have an API that tells you the current number animals categorized by color sizeand mammal - perhaps running down your street. You may create a counter like this as you watch them go by. JMeter testplan

So I then access the object I've created above like so. This is an absolutely trivial case of randomly generating the variables but the example is only trying to convey how one may interact with the objects created by the configuration element. However you extract the data, you can access and interact with the Prometheus metric you've created.

See the Prometheus Javadocs for more information on their API.

JMeter testplan

Which will expose a counter with all the appropriate labels.

JMeter testplan

Usage Tips

Skipping samplers or other elements

You can re-use metrics in multiple listeners so long as they're defined in the exact same way. There will be undefined behavior if two or more listeners have the same metric (the same metric name) with different configurations.

Here you see first_random_sampler and second_random_sampler in the labels of this metric, but you do not see want_to_skip, the thing that we're trying to skip. Note the composition of the test plan as shown at the top of this page

JMeter testplan

Visualization

This plugin has limited "out of the box" functionality because it gives you, the user, total control over what the metric names may be. That said, here's a sample dashboard given here in examples such that if you have a local prometheus/grafana stack you can a dashboard that looks something like this.

JMeter testplan

Properties you can override

Property default description
prometheus.port 9270 The port the http server will bind to
prometheus.delay 0 The delay (in seconds) the http server will wait before being destroyed
prometheus.save.threads true True or false value to save and collect jmeter thread metrics
prometheus.save.threads.name jmeter_threads The name of the metric describing jmeter threads
prometheus.save.jvm true Collect metrics from the JVM

Building

To build, simply maven package:

mvn clean package

This creates 2 jars, a shaded jar that has all the dependencies within it (this is the one you want) and the original jar. Both are in the target directory. Simply move the jar to your $JMETER_HOME/lib/ext directory as with any JMeter plugin and you're ready to go!

Feedback

Feel free to open issues against this project, even to ask questions.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.