This Nginx module features customizable counters shared by all worker processes and, optionally, by configured sets of virtual servers.
- Directives
- Sharing between virtual servers
- Collecting all counters in a single JSON object
- Reloading Nginx configuration
- Persistent counters
- Histograms
- Predefined counters
- An example
- Remarks on using location ifs and complex conditions
- See also
Normal counters are updated on the latest request's log phase. They can be declared on server, location, and location-if configuration levels. Their cumulative values (except for references to run-time variables) are merged through all the levels from the top to the bottom when Nginx reads configuration.
counter $cnt_name1 set 1;
counter $cnt_name2 inc $inc_cnt_name2;
counter $cnt_name2 undo;Variables $cnt_name1 and $cnt_name2 can be accessed elsewhere in the
configuration: they return values held in a shared memory and thus are equal
across all workers at the same moment. The second argument of the directive is
an operation — set, inc (i.e. increment), or undo. The third
argument is applicable to set and inc operations only. This is an optional
integer value (possibly negative) or a variable (possibly negated), the default
value is 1. The undo operation discards all changes to the counter made on
the upper levels of the merged hierarchies.
Starting from version 1.3 of the module, directive counter may declare
no-op counters such as
counter $cnt_name3;This is the exact equivalent of directive counter with option inc 0 which
can be used to declare variable $cnt_name3 as a counter with the appropriate
variable handler while avoiding access to the shared memory in the run-time.
Early counters are updated before rewrite directives and can be used to mark entry points before any rewrites and ifs. They are allowed only on location configuration level.
early_counter $cnt_name1 set 1;
early_counter $cnt_name2 inc $inc_cnt_name2;
early_counter $cnt_name2 undo;Meaning of the arguments corresponds to that of the normal counters.
Early counters are capable to update on every rewrite jump to another location. With the following configuration,
user nobody;
worker_processes 4;
events {
worker_connections 1024;
}
error_log /tmp/nginx-test-custom-counters-error.log warn;
http {
default_type application/octet-stream;
sendfile on;
access_log /tmp/nginx-test-custom-counters-access.log;
counters_survive_reload on;
server {
listen 8010;
server_name main;
location / {
early_counter $cnt_1 inc;
rewrite ^ /2;
}
location /2 {
early_counter $cnt_1 inc;
early_counter $cnt_2 inc;
rewrite ^ /3;
}
location /3 {
echo "cnt_1 = $cnt_1 | cnt_2 = $cnt_2";
}
}
}all early counters will be printed expectedly.
$ curl 'http://127.0.0.1:8010/'
cnt_1 = 2 | cnt_2 = 1
$ curl 'http://127.0.0.1:8010/'
cnt_1 = 4 | cnt_2 = 2A single counter can be declared both as normal and early if none of the merged location configuration hierarchies contains both types simultaneously.
Counters are shared between virtual servers if the latter have equal last
server names that form an identifier for the counter set. The counter set
identifier may also be declared explicitly using directive counter_set_id
which must precede all server's counters declarations.
When a counter is not mentioned within a virtual server being a member of some
other counter set, it gets unreachable in this virtual server. Unreachable
counters are displayed as empty strings, but this is configurable on main or
server configuration levels via directive display_unreachable_counter_as,
e.g.
display_unreachable_counter_as -;Starting from version 2.0 of the module, a new predefined variable
$cnt_collection can be used to collect values of all counters from all counter
sets and display them as a single JSON object. See an example.
Counters may survive after Nginx configuration reload, provided directive
counters_survive_reload was set on main or server configuration levels.
Counters from a specific counter set will not survive if their number in the
set has changed in the new configuration. Also avoid changing the order of
counters declarations, otherwise survived counters will pick values of their
mates that were standing on these positions before reloading.
Counters that survive reload may also be saved and loaded back when Nginx exits and starts respectively. To enjoy this feature, you need to add on http configuration level lines
counters_survive_reload on;
counters_persistent_storage /var/lib/nginx/counters.json 10s;The first directive can be moved inside server levels of the configuration
where counters persistency is really wanted. Path /var/lib/nginx/counters.json
denotes the directory where the counters will be saved. If the path is relative
(i.e. it does not start with /), then the counters will be saved in the
prefix directory: run nginx -h to see where default prefix directory is
located.
Value 10s defines time interval for saving persistent counters in a backup storage. This argument is optional: if not set then the counters won't be written into the backup storage. The name of the backup file corresponds to the name of the main persistent storage with suffix ~ added. The file gets written by a worker process when a user request comes to a virtual server associated with an existing counter set and the specified time interval from the last write has elapsed.
Writing to the backup storage can be useful to restore persistent counters on power outage or kill -9 of the Nginx master process. In such cases the main storage will be replaced by the backup storage automatically given that the latter has more recent modification time and is not corrupted.
Persistent counters require library JSMN,
which is header-only. It means that for building them, you need to put file
jsmn.h in the source directory of this module or in a standard system include
path such as /usr/include. If you want to enable building persistent counters,
set environment variable $NGX_HTTP_CUSTOM_COUNTERS_PERSISTENCY to y or yes
before or when running Nginx configure script, e.g.
$ NGX_HTTP_CUSTOM_COUNTERS_PERSISTENCY=yes ./configure ...Histograms provide a convenient way of dealing with a set of normal counters associated with an arbitrary range.
histogram $hst_name 12 $bound_var;
histogram $hst_name reuse;
histogram $hst_name undo;
histogram $hst_name reset;The upper line declares a histogram with 12 bins. The histogram must be bound
to a variable to read the number of the bin to increment from. In this example,
it's expected that variable $bound_var will return numbers in range 0 –
11 according to the number of the histogram bins. If it returns some unexpected
value then variable $hst_name_err (which was declared implicitly) will be
incremented instead of the range counters. The counters themselves and their
cumulative count value can be accessed directly via implicitly declared
variables $hst_name_00 .. $hst_name_11 and $hst_name_cnt. Notice that
rarely, when shown in variable $cnt_collection, the error and the count values
can be very slightly inconsistent in relation to the range counters: this may
happen because all counters get updated independently, and the updates may occur
in the middle of building of the collection when there are more than one worker
processes.
To simplify detection of the bin to increment in the case of a contiguous value
distribution, directive map_to_range_index can be used. For example,
map_to_range_index $request_time $request_time_bin
0.005
0.01
0.05;shall return in variable $request_time_bin values from 0 to 3 depending on
the value of variable $request_time: if the request time was less than or
equal to 0.005 then its value will be 0, otherwise, if the request time was
less than or equal to 0.01 then its value will be 1, and so later, finally,
if the request time was more than 0.05 then its value will be 3.
A histogram with the same name may be declared only once in a counter set. Sometimes it seems very restrictive, for example, when a histogram is supposed to collect data in two or more virtual servers. In this case, a histogram can be re-declared with directive reuse, given that it was already declared in this counter set in upper lines of the configuration.
Histograms layout can be observed via predefined variable $cnt_histograms.
There is a number of predefined counter variables: 7 counters from the Nginx
stub status module (available only when Nginx was compiled with option
--with-http_stub_status_module): $cnt_stub_status_accepted,
$cnt_stub_status_handled, $cnt_stub_status_requests,
$cnt_stub_status_active, $cnt_stub_status_reading,
$cnt_stub_status_writing, $cnt_stub_status_waiting, and 4 counters related
to the Nginx master process timing: $cnt_uptime and $cnt_uptime_reload which
contain the number of seconds elapsed since the start and the last reload of the
master process respectively, and $cnt_start_time and $cnt_start_time_reload
which contain the timestamps (in seconds elapsed since the start of the UNIX
Epoch) of the start and the last reload of the master process respectively.
All predefined counters are not associated with any counter set identifier, nor
are they collected in variable $cnt_collection.
user nobody;
worker_processes 4;
events {
worker_connections 1024;
}
error_log /tmp/nginx-test-custom-counters-error.log warn;
http {
default_type application/octet-stream;
sendfile on;
access_log /tmp/nginx-test-custom-counters-access.log;
map_to_range_index $request_time $request_time_bin
0.005
0.01
0.05
0.1
0.5
1.0
5.0
10.0
30.0
60.0;
counters_survive_reload on;
server {
listen 8010;
server_name main;
counter $cnt_all_requests inc;
set $inc_a_requests 0;
if ($arg_a) {
set $inc_a_requests 1;
}
location / {
return 200;
}
counter $cnt_a_requests inc $inc_a_requests;
counter $cnt_test1_requests inc;
counter $cnt_test2_requests inc;
counter $cnt_test3_requests inc;
location /test {
counter $cnt_test_requests inc;
if ($arg_a) {
counter $cnt_test_a_requests inc;
break;
}
if ($arg_b) {
counter $cnt_test_b_requests inc;
return 200;
}
echo "All requests before this: $cnt_all_requests";
}
location /test/rewrite {
early_counter $ecnt_test_requests inc;
rewrite ^ /test last;
}
counter $cnt_bytes_sent inc $bytes_sent;
}
server {
listen 8020;
server_name monitor.main;
counter_set_id main;
allow 127.0.0.1;
deny all;
location / {
echo -n "all = $cnt_all_requests";
echo -n " | all?a = $cnt_a_requests";
echo -n " | /test = $cnt_test_requests";
echo -n " | /test?a = $cnt_test_a_requests";
echo -n " | /test?b = $cnt_test_b_requests";
echo " | /test/rewrite = $ecnt_test_requests";
}
location ~* ^/reset/a/(\d+)$ {
set $set_a_requests $1;
counter $cnt_a_requests set $set_a_requests;
counter $cnt_test_a_requests set $set_a_requests;
return 200;
}
location /bytes_sent {
echo "bytes_sent = $cnt_bytes_sent";
}
location /all {
default_type application/json;
echo $cnt_collection;
}
location /histograms {
default_type application/json;
echo $cnt_histograms;
}
}
server {
listen 8030;
server_name other;
counter $cnt_test1_requests inc;
display_unreachable_counter_as -;
location / {
echo "all = $cnt_all_requests";
}
}
server {
listen 8040;
server_name test.histogram;
histogram $hst_request_time 11 $request_time_bin;
location / {
echo_sleep 0.5;
echo Ok;
}
location /1 {
echo_sleep 1;
echo Ok;
}
}
server {
listen 8050;
server_name monitor.test.histogram;
counter_set_id test.histogram;
location / {
echo "all bins: $hst_request_time";
echo "bin 04: $hst_request_time_04";
}
location /reset {
histogram $hst_request_time reset;
echo Ok;
}
}
}Let's run some curl tests.
$ curl 'http://127.0.0.1:8020/'
all = 0 | all?a = 0 | /test = 0 | /test?a = 0 | /test?b = 0 | /test/rewrite = 0
$ curl 'http://127.0.0.1:8010/test'
All requests before this: 0
$ curl 'http://127.0.0.1:8020/'
all = 1 | all?a = 0 | /test = 1 | /test?a = 0 | /test?b = 0 | /test/rewrite = 0
$ curl 'http://127.0.0.1:8010/?a=1'
$ curl 'http://127.0.0.1:8020/'
all = 2 | all?a = 1 | /test = 1 | /test?a = 0 | /test?b = 0 | /test/rewrite = 0
$ curl 'http://127.0.0.1:8010/test?b=1'
$ curl 'http://127.0.0.1:8020/'
all = 3 | all?a = 1 | /test = 2 | /test?a = 0 | /test?b = 1 | /test/rewrite = 0
$ curl 'http://127.0.0.1:8010/test?b=1&a=2'
All requests before this: 3
$ curl 'http://127.0.0.1:8020/'
all = 4 | all?a = 2 | /test = 3 | /test?a = 1 | /test?b = 1 | /test/rewrite = 0
$ curl 'http://127.0.0.1:8010/test/rewrite?b=1&a=2'
All requests before this: 4
$ curl 'http://127.0.0.1:8020/'
all = 5 | all?a = 3 | /test = 4 | /test?a = 2 | /test?b = 1 | /test/rewrite = 1
$ curl 'http://127.0.0.1:8020/reset/a/0'
$ curl 'http://127.0.0.1:8020/'
all = 5 | all?a = 0 | /test = 4 | /test?a = 0 | /test?b = 1 | /test/rewrite = 1
$ curl 'http://127.0.0.1:8020/reset/a/9'
$ curl 'http://127.0.0.1:8020/'
all = 5 | all?a = 9 | /test = 4 | /test?a = 9 | /test?b = 1 | /test/rewrite = 1Now let's see how many bytes were sent by Nginx so far.
$ curl 'http://127.0.0.1:8020/bytes_sent'
bytes_sent = 949And finally, let's see all counters at once.
$ curl -s 'http://127.0.0.1:8020/all' | jq
{
"main": {
"cnt_all_requests": 5,
"cnt_a_requests": 9,
"cnt_test1_requests": 5,
"cnt_test2_requests": 5,
"cnt_test3_requests": 5,
"cnt_test_requests": 4,
"cnt_test_a_requests": 9,
"cnt_test_b_requests": 1,
"ecnt_test_requests": 1,
"cnt_bytes_sent": 949
},
"other": {
"cnt_test1_requests": 0
},
"test.histogram": {
"hst_request_time_00": 0,
"hst_request_time_01": 0,
"hst_request_time_02": 0,
"hst_request_time_03": 0,
"hst_request_time_04": 0,
"hst_request_time_05": 0,
"hst_request_time_06": 0,
"hst_request_time_07": 0,
"hst_request_time_08": 0,
"hst_request_time_09": 0,
"hst_request_time_10": 0,
"hst_request_time_cnt": 0,
"hst_request_time_err": 0
}
}It's time to test our histogram.
$ for in in {1..20} ; do curl -D- 'http://localhost:8040/' & done
...
$ for in in {1..50} ; do curl -D- 'http://localhost:8040/1' & done
...Locations / and /1 in the virtual server test.histogram delay responses for 0.5 and 1 seconds respectively. We can check this from the values of the histogram counters.
$ curl -s 'http://127.0.0.1:8020/all' | jq {\"test.histogram\"}
{
"test.histogram": {
"hst_request_time_00": 0,
"hst_request_time_01": 0,
"hst_request_time_02": 0,
"hst_request_time_03": 0,
"hst_request_time_04": 13,
"hst_request_time_05": 45,
"hst_request_time_06": 12,
"hst_request_time_07": 0,
"hst_request_time_08": 0,
"hst_request_time_09": 0,
"hst_request_time_10": 0,
"hst_request_time_cnt": 70,
"hst_request_time_err": 0
}
}From this output, we can see that there were 70 requests spread in bins 04 – 06 which correspond approximately to a time range from 0.5 to 1-and-more seconds.
Let's see how to access all the bins at once and a specific bin.
$ curl 'http://127.0.0.1:8050/'
all bins: 0,0,0,0,13,45,12,0,0,0,0
bin 04: 13And we also have a way to reset the histogram.
$ curl -s 'http://127.0.0.1:8050/reset'
Ok
$ curl -s 'http://127.0.0.1:8020/all' | jq {\"test.histogram\"}
{
"test.histogram": {
"hst_request_time_00": 0,
"hst_request_time_01": 0,
"hst_request_time_02": 0,
"hst_request_time_03": 0,
"hst_request_time_04": 0,
"hst_request_time_05": 0,
"hst_request_time_06": 0,
"hst_request_time_07": 0,
"hst_request_time_08": 0,
"hst_request_time_09": 0,
"hst_request_time_10": 0,
"hst_request_time_cnt": 0,
"hst_request_time_err": 0
}
}Though is not present in this example, histogram operation undo disables changing the histogram in the scope where it is declared.
Finally, we can examine how all histograms lay out in counter sets.
$ curl -s 'http://127.0.0.1:8020/histograms' | jq
{
"main": {},
"other": {},
"test.histogram": {
"hst_request_time": {
"range": {
"hst_request_time_00": "0.005",
"hst_request_time_01": "0.01",
"hst_request_time_02": "0.05",
"hst_request_time_03": "0.1",
"hst_request_time_04": "0.5",
"hst_request_time_05": "1.0",
"hst_request_time_06": "5.0",
"hst_request_time_07": "10.0",
"hst_request_time_08": "30.0",
"hst_request_time_09": "60.0",
"hst_request_time_10": "+Inf"
},
"cnt": [
"hst_request_time_cnt",
"cnt"
],
"err": [
"hst_request_time_err",
"err"
]
}
}
}Originally in Nginx, location ifs were designed for a very special task:
replacing location configuration when a given condition matches, not for
doing anything. That's why using them for only checking a counter like when
testing against $arg_a in location /test is a bad idea in general. In
contrast, server ifs do not change location configurations and can be used for
checking increment or set values like $inc_a_requests. In our example we can
simply replace location if test
if ($arg_a) {
counter $cnt_test_a_requests inc;
break;
}with
counter $cnt_test_a_requests $inc_a_requests;However if condition testing in Nginx is not as powerful as it may be required. If you need a complex condition testing then consider binding increment or set variables to a full-featured programming language's handler. For example, let's increment a counter when a base64-encoded value contains a version tag like v=1. To make all required computations, let's use Haskell and Nginx Haskell module.
Put directive haskell compile with Haskell function hasVTag on http
configuration level.
haskell compile standalone /tmp/ngx_haskell.hs '
import Data.ByteString.Base64
import Data.Maybe
import Text.Regex.PCRE
hasVTag = either (const False) (matchTest r) . decode
where r = makeRegex "\\\\bv=\\\\d+\\\\b" :: Regex
NGX_EXPORT_B_Y (hasVTag)
';Then put the next 2 lines into location /test.
haskell_run hasVTag $hs_inc_cnt_vtag $cookie_misc;
counter $cnt_test_cookie_misc_vtag inc $hs_inc_cnt_vtag;Counter cnt_test_cookie_misc_vtag increments when value of cookie Misc matches against a version tag compiled as a regular expression with makeRegex.
By adding another line with echo
echo -n " | /test?misc:vtag = $cnt_test_cookie_misc_vtag";into location / in the second virtual server, the counter gets monitored along with other custom counters.
To test this, run
$ curl -b 'Misc=bW9kZT10ZXN0LHY9Mg==' 'http://localhost:8010/test'(value bW9kZT10ZXN0LHY9Mg== is base64-encoded string mode=test,v=2, try other variants with or without a version tag too), and watch the value of the counter cnt_test_cookie_misc_vtag.
- Универсальные счетчики в nginx: замечания к реализации модуля (in Russian). Remarks on implementation of the module.
- Haskell module NgxExport.Tools.Prometheus to convert custom counters from this module to Prometheus metrics.