Permalink
Fetching contributors…
Cannot retrieve contributors at this time
286 lines (206 sloc) 16.2 KB

varnishhoststat

Display to the statistics for each domain or url-pattern

Author: Shohei Tanaka(@xcir)
Date: 2017-11-08
Version: 52.20
Support Varnish Version:4.0.x, 4.1.x, 5.0.x, 5.1.x, 5.2.x
Manual section:1

For Varnish3.0.x

See this link. https://github.com/xcir/varnishHostStat/tree/varnish30

Versioning

[varnish-version].[library-version]

50.17 is v17 for Varnish5.0.x

DESCRIPTION

Display to the Statistics for each domain or url-pattern

ATTENTION

This script use to high cpu power. Is half cpu usage as compared with the varnishd in my environment.

SAMPLE

Group by host (varnishhoststat.py -i 5)

2014-02-03 23:13:00 - 2014-02-03 23:13:05 (interval:5)
Host                                               | Mbps        | rps         | hit         | time/req    | (H)time/req | (M)time/req | KB/req      | 2xx/s       | 3xx/s       | 4xx/s       | 5xx/s       |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
#alldata                                           | 0.000031    |    0.800000 |    0.000000 |    0.001085 |    0.000000 |    0.001085 |    0.004883 |    0.800000 |    0.000000 |    0.000000 |    0.000000 |
example.net                                        | 0.000015    |    0.400000 |    0.000000 |    0.000933 |    0.000000 |    0.000933 |    0.004883 |    0.400000 |    0.000000 |    0.000000 |    0.000000 |
hoge.example.net                                   | 0.000015    |    0.400000 |    0.000000 |    0.001237 |    0.000000 |    0.001237 |    0.004883 |    0.400000 |    0.000000 |    0.000000 |    0.000000 |

Group by url-pattern (varnishhoststat.py -i 5 -F "example.net@^/img/" -F "example.net@^/css/" -F "example.net")

2014-02-03 23:14:19 - 2014-02-03 23:14:23 (interval:5)
Host                                               | Mbps        | rps         | hit         | time/req    | (H)time/req | (M)time/req | KB/req      | 2xx/s       | 3xx/s       | 4xx/s       | 5xx/s       |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
#alldata                                           | 0.001808    |    1.800000 |   44.444444 |    0.000583 |    0.000119 |    0.000954 |    0.128581 |    1.000000 |    0.000000 |    0.800000 |    0.000000 |
[F1]example.net@^/img/                             | 0.000885    |    0.400000 |  100.000000 |    0.000110 |    0.000110 |    0.000000 |    0.283203 |    0.000000 |    0.000000 |    0.400000 |    0.000000 |
[F2]example.net@^/css/                             | 0.000885    |    0.400000 |  100.000000 |    0.000127 |    0.000127 |    0.000000 |    0.283203 |    0.000000 |    0.000000 |    0.400000 |    0.000000 |
[F3]example.net                                    | 0.000038    |    1.000000 |    0.000000 |    0.000954 |    0.000000 |    0.000954 |    0.004883 |    1.000000 |    0.000000 |    0.000000 |    0.000000 |

Raw-data output (varnishhoststat.py -r)

2014-02-03 23:26:48 - 2014-02-03 23:26:57 (interval:10)
Host                                               | req         | fetch       | fetch_time  | no_fetch_time | totallen    | 2xx         | 3xx         | 4xx         | 5xx         |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
#alldata                                           |          23 |          14 |    0.012728 |      0.000753 |        2680 |          14 |           0 |           9 |           0 |
example.net                                        |          18 |           9 |    0.008177 |      0.000753 |        2655 |           9 |           0 |           9 |           0 |
hoge.example.net                                   |           5 |           5 |    0.004551 |      0.000000 |          25 |           5 |           0 |           0 |           0 |

JSON-format output (varnishhoststat.py -j)

{"hoge.example.net": {"avg_fsize": 0.0048828125, "rps": 0.5, "avg_not_fetch_time": 0.0, "hit": 0.0, "avg_fetch_time": 0.00086789131164550777, "fetch_time": 0.0043394565582275391, "2xx": 5, "avg_2xx": 0.5, "mbps": 1.9073486328125e-05, "req": 5, "5xx": 0, "avg_3xx": 0.0, "no_fetch_time": 0, "totallen": 25, "4xx": 0, "3xx": 0, "avg_time": 0.00086789131164550777, "avg_5xx": 0.0, "fetch": 5, "avg_4xx": 0.0}, "#alldata": {"avg_fsize": 0.1162109375, "rps": 2.5, "avg_not_fetch_time": 8.5520744323730466e-05, "hit": 40.0, "avg_fetch_time": 0.0009458700815836589, "fetch_time": 0.014188051223754883, "2xx": 15, "avg_2xx": 1.5, "mbps": 0.002269744873046875, "req": 25, "5xx": 0, "avg_3xx": 0.0, "no_fetch_time": 0.00085520744323730469, "totallen": 2975, "4xx": 10, "3xx": 0, "avg_time": 0.00060173034667968753, "avg_5xx": 0.0, "fetch": 15, "avg_4xx": 1.0}, "example.net": {"avg_fsize": 0.14404296875, "rps": 2.0, "avg_not_fetch_time": 8.5520744323730466e-05, "hit": 50.0, "avg_fetch_time": 0.00098485946655273442, "fetch_time": 0.0098485946655273438, "2xx": 10, "avg_2xx": 1.0, "mbps": 0.00225067138671875, "req": 20, "5xx": 0, "avg_3xx": 0.0, "no_fetch_time": 0.00085520744323730469, "totallen": 2950, "4xx": 10, "3xx": 0, "avg_time": 0.00053519010543823242, "avg_5xx": 0.0, "fetch": 10, "avg_4xx": 1.0}, "@start-time": 1391437481, "@end-time": 1391437490, "@info": ""}
{"hoge.example.net": {"avg_fsize": 0.0048828125, "rps": 0.5, "avg_not_fetch_time": 0.0, "hit": 0.0, "avg_fetch_time": 0.00083451271057128902, "fetch_time": 0.0041725635528564453, "2xx": 5, "avg_2xx": 0.5, "mbps": 1.9073486328125e-05, "req": 5, "5xx": 0, "avg_3xx": 0.0, "no_fetch_time": 0, "totallen": 25, "4xx": 0, "3xx": 0, "avg_time": 0.00083451271057128902, "avg_5xx": 0.0, "fetch": 5, "avg_4xx": 0.0}, "#alldata": {"avg_fsize": 0.1162109375, "rps": 2.5, "avg_not_fetch_time": 8.2373619079589844e-05, "hit": 40.0, "avg_fetch_time": 0.00090791384379069009, "fetch_time": 0.013618707656860352, "2xx": 15, "avg_2xx": 1.5, "mbps": 0.002269744873046875, "req": 25, "5xx": 0, "avg_3xx": 0.0, "no_fetch_time": 0.00082373619079589844, "totallen": 2975, "4xx": 10, "3xx": 0, "avg_time": 0.00057769775390624999, "avg_5xx": 0.0, "fetch": 15, "avg_4xx": 1.0}, "example.net": {"avg_fsize": 0.14404296875, "rps": 2.0, "avg_not_fetch_time": 8.2373619079589844e-05, "hit": 50.0, "avg_fetch_time": 0.00094461441040039062, "fetch_time": 0.0094461441040039062, "2xx": 10, "avg_2xx": 1.0, "mbps": 0.00225067138671875, "req": 20, "5xx": 0, "avg_3xx": 0.0, "no_fetch_time": 0.00082373619079589844, "totallen": 2950, "4xx": 10, "3xx": 0, "avg_time": 0.00051349401473999023, "avg_5xx": 0.0, "fetch": 10, "avg_4xx": 1.0}, "@start-time": 1391437491, "@end-time": 1391437500, "@info": ""}

Raw-data output by JSON-format(varnishhoststat.py -r -j)

{"hoge.example.net": {"fetch_time": 0.0037126541137695312, "2xx": 4, "req": 4, "5xx": 0, "no_fetch_time": 0, "totallen": 20, "4xx": 0, "3xx": 0, "fetch": 4}, "#alldata": {"fetch_time": 0.01218414306640625, "2xx": 13, "req": 23, "5xx": 0, "no_fetch_time": 0.00090909004211425781, "totallen": 2965, "4xx": 10, "3xx": 0, "fetch": 13}, "example.net": {"fetch_time": 0.0084714889526367188, "2xx": 9, "req": 19, "5xx": 0, "no_fetch_time": 0.00090909004211425781, "totallen": 2945, "4xx": 10, "3xx": 0, "fetch": 9}, "@start-time": 1391437527, "@end-time": 1391437536, "@info": ""}
{"hoge.example.net": {"fetch_time": 0.0052282810211181641, "2xx": 5, "req": 5, "5xx": 0, "no_fetch_time": 0, "totallen": 25, "4xx": 0, "3xx": 0, "fetch": 5}, "#alldata": {"fetch_time": 0.013852119445800781, "2xx": 15, "req": 25, "5xx": 0, "no_fetch_time": 0.00098705291748046875, "totallen": 2975, "4xx": 10, "3xx": 0, "fetch": 15}, "example.net": {"fetch_time": 0.0086238384246826172, "2xx": 10, "req": 20, "5xx": 0, "no_fetch_time": 0.00098705291748046875, "totallen": 2950, "4xx": 10, "3xx": 0, "fetch": 10}, "@start-time": 1391437537, "@end-time": 1391437546, "@info": ""}

OUTPUT FORMAT

-Default

[start-time] - [end-time] (interval:[interval time]) [error message]
Host                    | Mbps                 | rps             | hit      | time/req              | (H)time/req                             | (M)time/req                            | KB/req                     | 2xx/s                 | 3xx/s                 | 4xx/s                 | 5xx/s
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Host or filter rule     | Traffic              | Request Per Sec | Hit per  | Average response time | Average response time by hit request    |  Average response time by miss request | average response size | HTTP status 2xx rate  | HTTP status 3xx rate  | HTTP status 4xx rate  | HTTP status 5xx rate

* hit rate
  Decision as follows:
  Fetch to backend = Miss
  Not fetch        = Hit

-Raw
[start-time] - [end-time] (interval:[interval time]) [error message]
Host                 | req           | fetch       | fetch_time                   | no_fetch_time         | totallen                 | 2xx                    | 3xx                    | 4xx                    | 5xx                    |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
Host or filter rule  | Request count | Fetch count | Total time(fetch to backend) | Total time(not fetch) | Total transfer size | HTTP-status count(2xx) | HTTP-status count(3xx) | HTTP-status count(4xx) | HTTP-status count(5xx) |

OPTION

-r -j -i [interval] -a -F [filter pattern] -R [replace pattern] -f [field name(default:host)] --status [status,status,...] --start [second] --sopath [libvarnishapi.so] -w [file-name] -D -n [instance-name] -P [pid-file] -V

-r

Raw data(no summarize)

-j

Output json format

-i [interval]

Specify interval second. Default is 10 second.

example

#10 second
-i 10

-D

Daemonize.

-P [pid-file]

Write the process's PID to the specified file.(require -D option)

-F [host@url-pattern]

Specify filter pattern. Statistics for each domain separately ,if you do not specified.

example

#Filter by example.net (ends-with match)
#This pattern is match to a.example.net and b.example.net and example.net
-F example.net

#Filter by example.net^/img/[0-9]
#This pattern is match to a.example.net/img/0 and b.example.net/img/1 and example.net/img/2
-F "example.net@^/img/[0-9]"

#Filter by example.net^/img/[0-9] and other example.net
-F "example.net@^/img/[0-9]" -F example.net

#Bad pattern
#Not match to example.net@^/img/[0-9]
-F example.net -F "example.net@^/img/[0-9]"

-R [pattern/replace]

Specify replace pattern for the host header.

example

#Replace from [.] to [-].
#This pattern is replace from a.example.net to a-example-net
-R "\./-"

#Using group (see: https://docs.python.org/2.7/library/re.html )
-R "(example|varnish-cache).*/\1"

-f [field name(default:host)]

Specify field name. Default field is "host" This option not change output field name(Host)

example

#./varnishhoststat.py -f user-agent
2015-12-29 01:23:38 - 2015-12-29 01:23:47 (interval:10)
Host                                               | Mbps        | rps         | hit         | time/req    | (H)time/req | (M)time/req | KB/req      | 2xx/s       | 3xx/s       | 4xx/s       | 5xx/s       |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
#alldata                                           | 0.001697    |    0.400000 |    0.000000 |    0.000032 |    0.000000 |    0.000032 |    0.542969 |    0.000000 |    0.000000 |    0.000000 |    0.400000 |
Wget/1.15 (linux-gnu)                              | 0.000845    |    0.200000 |    0.000000 |    0.000034 |    0.000000 |    0.000034 |    0.541016 |    0.000000 |    0.000000 |    0.000000 |    0.200000 |
curl/7.35.0                                        | 0.000851    |    0.200000 |    0.000000 |    0.000029 |    0.000000 |    0.000029 |    0.544922 |    0.000000 |    0.000000 |    0.000000 |    0.200000 |

-a

Additional filter. Show to statistics for each domain and filtered statistics.

-w [file-name]

Specify write log file-name. Move log file ,if you want rotation. (Don't send HUP)

-V

Show version info.

--status [status,status,...]

Append summarize status

example

#./varnishhoststat.py -status 200,404
2016-12-30 15:33:40 - 2016-12-30 15:33:49 (interval:10)
Host                                               | Mbps        | rps         | hit         | time/req    | (H)time/req | (M)time/req | KB/req      | 2xx/s       | 3xx/s       | 4xx/s       | 5xx/s       | 200/s       | 404/s       |
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
#alldata                                           | 0.030865    |    1.200000 |    0.000000 |    0.000052 |    0.000000 |    0.000052 |    3.292236 |    1.100000 |    0.000000 |    0.100000 |    0.000000 |    1.100000 |    0.100000 |
192.168.1.37:6081                                  | 0.030865    |    1.200000 |    0.000000 |    0.000052 |    0.000000 |    0.000052 |    3.292236 |    1.100000 |    0.000000 |    0.100000 |    0.000000 |    1.100000 |    0.100000 |

--start [second]

Fix starting time.

--sopath [libvarnishapi.so]

Specify the libvarnishapi.so path, if you non-standard path.

-n [instance-name]

Select which named Varnishd instance to use in multi-instance set-ups. (See -n for varnishstat, etc)

HISTORY

Version 52.20: Support Varnish5.2.x. Enhance perfomance.

Version 50.19: Fix if it's not come request within the interval-time, get an error. Support --status option.

Version 50.18: Use Python3. Enhanced performance. add @info (output sample:Log overrun, Log abandoned)

Version 50.17: Update varnishapi(include 50.18)

Version 0.16-varnish40: Update varnishapi(include 40.16)

Version 0.15-varnish40: Update varnishapi(include 40.15)

Version 0.14-varnish40: Update varnishapi(include 40.14)

Version 0.13-varnish40: Initial support for Python3. Feedback is welcome.

Version 0.12-varnish40: Support -f, -R option

Version 0.11-varnish40: Fix Crash if log abandoned.

Version 0.10-varnish40: Support --sopath option.

Version 0.9-varnish40: Support change to Varnish4(In development. Don't use production)

Version 0.8-varnish30: Support -n option. thx mrmonkington[#1]

Version 0.7-varnish30: Reopen VSM ,if Varnish restarted.

Version 0.6-varnish30: Support -a option

Version 0.5-varnish30: Support -D -P option

Version 0.4-varnish30: Support -w option

Version 0.3-varnish30: Support --start option, Bugfix

Version 0.2-varnish30: Support -r -j option

Version 0.1-varnish30: First version