varnish: collect more metrics

[ilyam8]

Feature idea summary

varnish module is outdated

New metrics were added in v6.0(link) (some people still use v4)

Besides that

varnishstat -1

returns a lot of metrics and we are collecting only few of them 😢

ver 5.0.0 example (clean install, 2 backends, 279 metrics)

MAIN.uptime               4554         1.00 Child process uptime
MAIN.sess_conn               3         0.00 Sessions accepted
MAIN.sess_drop               0         0.00 Sessions dropped
MAIN.sess_fail               0         0.00 Session accept failures
MAIN.client_req_400            0         0.00 Client requests received, subject to 400 errors
MAIN.client_req_417            0         0.00 Client requests received, subject to 417 errors
MAIN.client_req               13         0.00 Good client requests received
MAIN.cache_hit                 0         0.00 Cache hits
MAIN.cache_hitpass             0         0.00 Cache hits for pass
MAIN.cache_miss                0         0.00 Cache misses
MAIN.backend_conn              2         0.00 Backend conn. success
MAIN.backend_unhealthy            0         0.00 Backend conn. not attempted
MAIN.backend_busy                 0         0.00 Backend conn. too many
MAIN.backend_fail                 0         0.00 Backend conn. failures
MAIN.backend_reuse               11         0.00 Backend conn. reuses
MAIN.backend_recycle             13         0.00 Backend conn. recycles
MAIN.backend_retry                0         0.00 Backend conn. retry
MAIN.fetch_head                   0         0.00 Fetch no body (HEAD)
MAIN.fetch_length                 0         0.00 Fetch with Length
MAIN.fetch_chunked                2         0.00 Fetch chunked
MAIN.fetch_eof                    0         0.00 Fetch EOF
MAIN.fetch_bad                    0         0.00 Fetch bad T-E
MAIN.fetch_none                   0         0.00 Fetch no body
MAIN.fetch_1xx                    0         0.00 Fetch no body (1xx)
MAIN.fetch_204                    0         0.00 Fetch no body (204)
MAIN.fetch_304                   11         0.00 Fetch no body (304)
MAIN.fetch_failed                 0         0.00 Fetch failed (all causes)
MAIN.fetch_no_thread              0         0.00 Fetch failed (no thread)
MAIN.pools                        2          .   Number of thread pools
MAIN.threads                    200          .   Total number of threads
MAIN.threads_limited              0         0.00 Threads hit max
MAIN.threads_created            200         0.04 Threads created
MAIN.threads_destroyed            0         0.00 Threads destroyed
MAIN.threads_failed               0         0.00 Thread creation failed
MAIN.thread_queue_len             0          .   Length of session queue
MAIN.busy_sleep                   0         0.00 Number of requests sent to sleep on busy objhdr
MAIN.busy_wakeup                  0         0.00 Number of requests woken after sleep on busy objhdr
MAIN.busy_killed                  0         0.00 Number of requests killed after sleep on busy objhdr
MAIN.sess_queued                  0         0.00 Sessions queued for thread
MAIN.sess_dropped                 0         0.00 Sessions dropped for thread
MAIN.n_object                     0          .   object structs made
MAIN.n_vampireobject              0          .   unresurrected objects
MAIN.n_objectcore                 0          .   objectcore structs made
MAIN.n_objecthead                 0          .   objecthead structs made
MAIN.n_backend                    2          .   Number of backends
MAIN.n_expired                    0          .   Number of expired objects
MAIN.n_lru_nuked                  0          .   Number of LRU nuked objects
MAIN.n_lru_moved                  0          .   Number of LRU moved objects
MAIN.losthdr                      0         0.00 HTTP header overflows
MAIN.s_sess                       3         0.00 Total sessions seen
MAIN.s_req                       13         0.00 Total requests seen
MAIN.s_pipe                       0         0.00 Total pipe sessions seen
MAIN.s_pass                      13         0.00 Total pass-ed requests seen
MAIN.s_fetch                     13         0.00 Total backend fetches initiated
MAIN.s_synth                      0         0.00 Total synthethic responses made
MAIN.s_req_hdrbytes            7507         1.65 Request header bytes
MAIN.s_req_bodybytes              0         0.00 Request body bytes
MAIN.s_resp_hdrbytes           3180         0.70 Response header bytes
MAIN.s_resp_bodybytes           573         0.13 Response body bytes
MAIN.s_pipe_hdrbytes              0         0.00 Pipe request header bytes
MAIN.s_pipe_in                    0         0.00 Piped bytes from client
MAIN.s_pipe_out                   0         0.00 Piped bytes to client
MAIN.sess_closed                  0         0.00 Session Closed
MAIN.sess_closed_err              3         0.00 Session Closed with error
MAIN.sess_readahead               0         0.00 Session Read Ahead
MAIN.sess_herd                   15         0.00 Session herd
MAIN.sc_rem_close                 0         0.00 Session OK  REM_CLOSE
MAIN.sc_req_close                 0         0.00 Session OK  REQ_CLOSE
MAIN.sc_req_http10                0         0.00 Session Err REQ_HTTP10
MAIN.sc_rx_bad                    0         0.00 Session Err RX_BAD
MAIN.sc_rx_body                   0         0.00 Session Err RX_BODY
MAIN.sc_rx_junk                   0         0.00 Session Err RX_JUNK
MAIN.sc_rx_overflow               0         0.00 Session Err RX_OVERFLOW
MAIN.sc_rx_timeout                3         0.00 Session Err RX_TIMEOUT
MAIN.sc_tx_pipe                   0         0.00 Session OK  TX_PIPE
MAIN.sc_tx_error                  0         0.00 Session Err TX_ERROR
MAIN.sc_tx_eof                    0         0.00 Session OK  TX_EOF
MAIN.sc_resp_close                0         0.00 Session OK  RESP_CLOSE
MAIN.sc_overload                  0         0.00 Session Err OVERLOAD
MAIN.sc_pipe_overflow             0         0.00 Session Err PIPE_OVERFLOW
MAIN.sc_range_short               0         0.00 Session Err RANGE_SHORT
MAIN.shm_records               4268         0.94 SHM records
MAIN.shm_writes                3091         0.68 SHM writes
MAIN.shm_flushes                  0         0.00 SHM flushes due to overflow
MAIN.shm_cont                     0         0.00 SHM MTX contention
MAIN.shm_cycles                   0         0.00 SHM cycles through buffer
MAIN.backend_req                 13         0.00 Backend requests made
MAIN.n_vcl                        1         0.00 Number of loaded VCLs in total
MAIN.n_vcl_avail                  1         0.00 Number of VCLs available
MAIN.n_vcl_discard                0         0.00 Number of discarded VCLs
MAIN.bans                         1          .   Count of bans
MAIN.bans_completed               1          .   Number of bans marked 'completed'
MAIN.bans_obj                     0          .   Number of bans using obj.*
MAIN.bans_req                     0          .   Number of bans using req.*
MAIN.bans_added                   1         0.00 Bans added
MAIN.bans_deleted                 0         0.00 Bans deleted
MAIN.bans_tested                  0         0.00 Bans tested against objects (lookup)
MAIN.bans_obj_killed              0         0.00 Objects killed by bans (lookup)
MAIN.bans_lurker_tested            0         0.00 Bans tested against objects (lurker)
MAIN.bans_tests_tested             0         0.00 Ban tests tested against objects (lookup)
MAIN.bans_lurker_tests_tested            0         0.00 Ban tests tested against objects (lurker)
MAIN.bans_lurker_obj_killed              0         0.00 Objects killed by bans (lurker)
MAIN.bans_dups                           0         0.00 Bans superseded by other bans
MAIN.bans_lurker_contention              0         0.00 Lurker gave way for lookup
MAIN.bans_persisted_bytes               16          .   Bytes used by the persisted ban lists
MAIN.bans_persisted_fragmentation            0          .   Extra bytes in persisted ban lists due to fragmentation
MAIN.n_purges                                0          .   Number of purge operations executed
MAIN.n_obj_purged                            0          .   Number of purged objects
MAIN.exp_mailed                              0         0.00 Number of objects mailed to expiry thread
MAIN.exp_received                            0         0.00 Number of objects received by expiry thread
MAIN.hcb_nolock                              0         0.00 HCB Lookups without lock
MAIN.hcb_lock                                0         0.00 HCB Lookups with lock
MAIN.hcb_insert                              0         0.00 HCB Inserts
MAIN.esi_errors                              0         0.00 ESI parse errors (unlock)
MAIN.esi_warnings                            0         0.00 ESI parse warnings (unlock)
MAIN.vmods                                   1          .   Loaded VMODs
MAIN.n_gzip                                  0         0.00 Gzip operations
MAIN.n_gunzip                                2         0.00 Gunzip operations
MAIN.vsm_free                           973664          .   Free VSM space
MAIN.vsm_used                         83960944          .   Used VSM space
MAIN.vsm_cooling                             0          .   Cooling VSM space
MAIN.vsm_overflow                            0          .   Overflow VSM space
MAIN.vsm_overflowed                          0         0.00 Overflowed VSM space
MGT.uptime                                4554         1.00 Management process uptime
MGT.child_start                              1         0.00 Child process started
MGT.child_exit                               0         0.00 Child process normal exit
MGT.child_stop                               0         0.00 Child process unexpected exit
MGT.child_died                               0         0.00 Child process died (signal)
MGT.child_dump                               0         0.00 Child process core dumped
MGT.child_panic                              0         0.00 Child process panic
MEMPOOL.busyobj.live                         0          .   In use
MEMPOOL.busyobj.pool                        10          .   In Pool
MEMPOOL.busyobj.sz_wanted                65536          .   Size requested
MEMPOOL.busyobj.sz_actual                65504          .   Size allocated
MEMPOOL.busyobj.allocs                      13         0.00 Allocations
MEMPOOL.busyobj.frees                       13         0.00 Frees
MEMPOOL.busyobj.recycle                     13         0.00 Recycled from pool
MEMPOOL.busyobj.timeout                      0         0.00 Timed out from pool
MEMPOOL.busyobj.toosmall                     0         0.00 Too small to recycle
MEMPOOL.busyobj.surplus                      0         0.00 Too many for pool
MEMPOOL.busyobj.randry                       0         0.00 Pool ran dry
MEMPOOL.req0.live                            0          .   In use
MEMPOOL.req0.pool                           10          .   In Pool
MEMPOOL.req0.sz_wanted                   65536          .   Size requested
MEMPOOL.req0.sz_actual                   65504          .   Size allocated
MEMPOOL.req0.allocs                         13         0.00 Allocations
MEMPOOL.req0.frees                          13         0.00 Frees
MEMPOOL.req0.recycle                        13         0.00 Recycled from pool
MEMPOOL.req0.timeout                         1         0.00 Timed out from pool
MEMPOOL.req0.toosmall                        0         0.00 Too small to recycle
MEMPOOL.req0.surplus                         0         0.00 Too many for pool
MEMPOOL.req0.randry                          0         0.00 Pool ran dry
MEMPOOL.sess0.live                           0          .   In use
MEMPOOL.sess0.pool                          10          .   In Pool
MEMPOOL.sess0.sz_wanted                    512          .   Size requested
MEMPOOL.sess0.sz_actual                    480          .   Size allocated
MEMPOOL.sess0.allocs                         1         0.00 Allocations
MEMPOOL.sess0.frees                          1         0.00 Frees
MEMPOOL.sess0.recycle                        1         0.00 Recycled from pool
MEMPOOL.sess0.timeout                        1         0.00 Timed out from pool
MEMPOOL.sess0.toosmall                       0         0.00 Too small to recycle
MEMPOOL.sess0.surplus                        0         0.00 Too many for pool
MEMPOOL.sess0.randry                         0         0.00 Pool ran dry
MEMPOOL.req1.live                            0          .   In use
MEMPOOL.req1.pool                           10          .   In Pool
MEMPOOL.req1.sz_wanted                   65536          .   Size requested
MEMPOOL.req1.sz_actual                   65504          .   Size allocated
MEMPOOL.req1.allocs                          2         0.00 Allocations
MEMPOOL.req1.frees                           2         0.00 Frees
MEMPOOL.req1.recycle                         2         0.00 Recycled from pool
MEMPOOL.req1.timeout                         0         0.00 Timed out from pool
MEMPOOL.req1.toosmall                        0         0.00 Too small to recycle
MEMPOOL.req1.surplus                         0         0.00 Too many for pool
MEMPOOL.req1.randry                          0         0.00 Pool ran dry
MEMPOOL.sess1.live                           0          .   In use
MEMPOOL.sess1.pool                          10          .   In Pool
MEMPOOL.sess1.sz_wanted                    512          .   Size requested
MEMPOOL.sess1.sz_actual                    480          .   Size allocated
MEMPOOL.sess1.allocs                         2         0.00 Allocations
MEMPOOL.sess1.frees                          2         0.00 Frees
MEMPOOL.sess1.recycle                        2         0.00 Recycled from pool
MEMPOOL.sess1.timeout                        2         0.00 Timed out from pool
MEMPOOL.sess1.toosmall                       0         0.00 Too small to recycle
MEMPOOL.sess1.surplus                        0         0.00 Too many for pool
MEMPOOL.sess1.randry                         0         0.00 Pool ran dry
SMA.s0.c_req                                 0         0.00 Allocator requests
SMA.s0.c_fail                                0         0.00 Allocator failures
SMA.s0.c_bytes                               0         0.00 Bytes allocated
SMA.s0.c_freed                               0         0.00 Bytes freed
SMA.s0.g_alloc                               0          .   Allocations outstanding
SMA.s0.g_bytes                               0          .   Bytes outstanding
SMA.s0.g_space                       268435456          .   Bytes available
SMA.Transient.c_req                         17         0.00 Allocator requests
SMA.Transient.c_fail                         0         0.00 Allocator failures
SMA.Transient.c_bytes                    37037         8.13 Bytes allocated
SMA.Transient.c_freed                    37037         8.13 Bytes freed
SMA.Transient.g_alloc                        0          .   Allocations outstanding
SMA.Transient.g_bytes                        0          .   Bytes outstanding
SMA.Transient.g_space                        0          .   Bytes available
VBE.boot.default.happy                       0          .   Happy health probes
VBE.boot.default.bereq_hdrbytes           4243         0.93 Request header bytes
VBE.boot.default.bereq_bodybytes             0         0.00 Request body bytes
VBE.boot.default.beresp_hdrbytes          1329         0.29 Response header bytes
VBE.boot.default.beresp_bodybytes          384         0.08 Response body bytes
VBE.boot.default.pipe_hdrbytes               0         0.00 Pipe request header bytes
VBE.boot.default.pipe_out                    0         0.00 Piped bytes to backend
VBE.boot.default.pipe_in                     0         0.00 Piped bytes from backend
VBE.boot.default.conn                        0          .   Concurrent connections to backend
VBE.boot.default.req                         7         0.00 Backend requests sent
VBE.boot.nginx2.happy                        0          .   Happy health probes
VBE.boot.nginx2.bereq_hdrbytes            3559         0.78 Request header bytes
VBE.boot.nginx2.bereq_bodybytes              0         0.00 Request body bytes
VBE.boot.nginx2.beresp_hdrbytes           1086         0.24 Response header bytes
VBE.boot.nginx2.beresp_bodybytes           189         0.04 Response body bytes
VBE.boot.nginx2.pipe_hdrbytes                0         0.00 Pipe request header bytes
VBE.boot.nginx2.pipe_out                     0         0.00 Piped bytes to backend
VBE.boot.nginx2.pipe_in                      0         0.00 Piped bytes from backend
VBE.boot.nginx2.conn                         0          .   Concurrent connections to backend
VBE.boot.nginx2.req                          6         0.00 Backend requests sent
LCK.backend.creat                            3         0.00 Created locks
LCK.backend.destroy                          0         0.00 Destroyed locks
LCK.backend.locks                           27         0.01 Lock Operations
LCK.backend_tcp.creat                        2         0.00 Created locks
LCK.backend_tcp.destroy                      0         0.00 Destroyed locks
LCK.backend_tcp.locks                       50         0.01 Lock Operations
LCK.ban.creat                                1         0.00 Created locks
LCK.ban.destroy                              0         0.00 Destroyed locks
LCK.ban.locks                              296         0.06 Lock Operations
LCK.busyobj.creat                           13         0.00 Created locks
LCK.busyobj.destroy                         13         0.00 Destroyed locks
LCK.busyobj.locks                           49         0.01 Lock Operations
LCK.cli.creat                                1         0.00 Created locks
LCK.cli.destroy                              0         0.00 Destroyed locks
LCK.cli.locks                             1529         0.34 Lock Operations
LCK.exp.creat                                1         0.00 Created locks
LCK.exp.destroy                              0         0.00 Destroyed locks
LCK.exp.locks                             1451         0.32 Lock Operations
LCK.hcb.creat                                1         0.00 Created locks
LCK.hcb.destroy                              0         0.00 Destroyed locks
LCK.hcb.locks                               26         0.01 Lock Operations
LCK.lru.creat                                2         0.00 Created locks
LCK.lru.destroy                              0         0.00 Destroyed locks
LCK.lru.locks                                0         0.00 Lock Operations
LCK.mempool.creat                            5         0.00 Created locks
LCK.mempool.destroy                          0         0.00 Destroyed locks
LCK.mempool.locks                        20285         4.45 Lock Operations
LCK.objhdr.creat                             1         0.00 Created locks
LCK.objhdr.destroy                           0         0.00 Destroyed locks
LCK.objhdr.locks                           117         0.03 Lock Operations
LCK.pipestat.creat                           1         0.00 Created locks
LCK.pipestat.destroy                         0         0.00 Destroyed locks
LCK.pipestat.locks                           0         0.00 Lock Operations
LCK.sess.creat                               3         0.00 Created locks
LCK.sess.destroy                             3         0.00 Destroyed locks
LCK.sess.locks                              29         0.01 Lock Operations
LCK.vbe.creat                                1         0.00 Created locks
LCK.vbe.destroy                              0         0.00 Destroyed locks
LCK.vbe.locks                             1522         0.33 Lock Operations
LCK.vcapace.creat                            1         0.00 Created locks
LCK.vcapace.destroy                          0         0.00 Destroyed locks
LCK.vcapace.locks                            0         0.00 Lock Operations
LCK.vcl.creat                                1         0.00 Created locks
LCK.vcl.destroy                              0         0.00 Destroyed locks
LCK.vcl.locks                               32         0.01 Lock Operations
LCK.vxid.creat                               1         0.00 Created locks
LCK.vxid.destroy                             0         0.00 Destroyed locks
LCK.vxid.locks                               2         0.00 Lock Operations
LCK.waiter.creat                             2         0.00 Created locks
LCK.waiter.destroy                           0         0.00 Destroyed locks
LCK.waiter.locks                           190         0.04 Lock Operations
LCK.wq.creat                                 3         0.00 Created locks
LCK.wq.destroy                               0         0.00 Destroyed locks
LCK.wq.locks                              6865         1.51 Lock Operations
LCK.wstat.creat                              1         0.00 Created locks
LCK.wstat.destroy                            0         0.00 Destroyed locks
LCK.wstat.locks                           1504         0.33 Lock Operations
LCK.sma.creat                                2         0.00 Created locks
LCK.sma.destroy                              0         0.00 Destroyed locks
LCK.sma.locks                               34         0.01 Lock Operations

it would be nice to collect much more!

Metrics description:
https://varnish-cache.org/docs/trunk/reference/varnish-counters.html

It is not easy to understand which of them are important and how to group them (this shoud be done for all versions started from 4). Help from varnish users highly appreciated.

Expected behavior

varnish module collects more metrics

[ilyam8]

@ernestojpg hey 👋

I see you are interested in varnish collector and uses Commercial/Enterprise version.

Current python implementation was written long time ago and not good: number of collected metrics, grouping, etc.

If you have time and willing to help it would be very nice 🙇‍♂️

[ernestojpg]

Hello @ilyam8 !

Yes, I could add more metrics easily. Do you want me to propose what would be the more relevant metrics?

[ilyam8]

@ernestojpg

Do you want me to propose what would be the more relevant metrics?

This. I feel like we need to forget that we have python version and start from the beginning:

• use latest stable version (both free and enterprise version)
• get list of metrics, compare with metrics docs
• go through the list and find everything we want to collect
• group metrics into charts
• think about alarms

---

Quick google search:

• https://www.section.io/docs/modules/varnish-cache/how-tos/varnish-metrics/

  • Main Counters
  • Lock Counters
  • Per Malloc Storage Counters
  • Per File Storage Counters
  • Per Backend Counters

• https://varnish-cache.org/docs/trunk/reference/varnish-counters.html

  • MAIN – Main counters
  • MGT – Management Process Counters
  • MEMPOOL – Memory Pool Counters
  • SMA – Malloc Stevedore Counters (what's Stevedore? 🤔 )
  • SMU – Umem Stevedore Counters
  • SMF – File Stevedore Counters
  • VBE – Backend Counters
  • LCK – Lock Counters

---

Ideally we need to go through every section (MAIN, MGT, ...) and try to understand: what the metric is, do we want to collect it, etc.

[ilyam8]

Closing this feature request and moving it to our community forum for further discussion (link will be posted soon).