Clear server counters#19
Open
alexanderstephan wants to merge 5 commits into
Open
Conversation
alexanderstephan
force-pushed
the
clear-counters-server
branch
4 times, most recently
from
July 15, 2026 15:14
8eb86a2 to
604ea0a
Compare
alexanderstephan
force-pushed
the
clear-counters-server
branch
from
July 16, 2026 13:31
604ea0a to
086e993
Compare
alexanderstephan
force-pushed
the
clear-counters-server
branch
from
July 17, 2026 08:05
086e993 to
b108db5
Compare
proxy_stats_clear_counters() did a blanket memset() on the be_counters / fe_counters / listener counter structs when clearing all counters. This zeroed the embedded shared.tg pointer array, which every hot path dereferences with no NULL check (for example _HA_ATOMIC_INC(&srv->counters.shared.tg[tgid-1]->cum_lbconn) in process_srv_queue()). After 'clear counters all' the next request on that server would segfault. The memset pattern predates the switch to per-thread-group shared counters (commit 5495c88 "MEDIUM: counters: Dynamically allocate per-thread group counters") and wasn't updated when shared.tg gained pointer-array semantics. Additionally, memset'ing only the outer struct never actually reset the accumulated counters in the pointed-at per-tgroup structs, so 'clear counters all' silently failed to match its documented "same effect as restarting" behaviour for cumulative counters (bytes, sessions, requests, ...). Introduce two helpers in src/counters.c: void counters_fe_reset(struct fe_counters *counters); void counters_be_reset(struct be_counters *counters); Both iterate shared.tg[0 .. nbtgroups-1] and zero the *contents* of each per-tgroup struct via memset, then zero the local (non-shared) fields of the outer struct. The shared.tg pointer array, each shared.tg[it] pointer, and shared.flags (COUNTERS_SHARED_F_LOCAL is set at boot and reflects allocation ownership, not counter state) are preserved. Refactor proxy_stats_clear_counters() to call these helpers instead of the inline memset. This fixes both the segfault after 'clear counters all' and the silent no-op on cumulative counters, and provides a shared primitive for a subsequent 'clear counters server <b>/<s>' command. In SHM stats-file mode, zeroing the per-tgroup structs affects every process attached to the same object; this matches the intended "reset everywhere" semantic of clear counters and is unchanged from the outer-memset intent. This bug should be backported wherever the per-thread-group shared counter refactor is present.
…unters
The plain "clear counters" command (OPER level) resets only the max/peak
gauges of each object's counters, leaving cumulative counters intact,
while "clear counters all" (ADMIN) performs a full reset via the
counters_{fe,be}_reset() helpers introduced earlier in this series.
The max-only reset was open-coded inline in proxy_stats_clear_counters()
with a hand-maintained list of fields, duplicated across the backend,
server and listener branches. This is fragile: a new max gauge added to
struct {fe,be}_counters can silently be forgotten here.
Factor the max-only reset into counters_fe_reset_max() and
counters_be_reset_max(), living next to the existing reset helpers, and
call them from proxy_stats_clear_counters(). Unlike the full-reset
helpers these cannot use a blanket memset because the max fields are
interleaved with live cumulative fields, so the gauges are zeroed
individually.
While consolidating, the two open-coded lists turned out to disagree:
the backend branch cleared conn_max / cps_max / rps_max but not
cur_sess_max, while the server branch cleared cur_sess_max but none of
the former. Both used the same struct be_counters, so these were latent
gaps rather than intentional differences. counters_be_reset_max() now
clears the union of all peak gauges, so plain "clear counters"
consistently resets every max gauge for both backends and servers.
No functional change for the full-reset ("all") path.
alexanderstephan
force-pushed
the
clear-counters-server
branch
from
July 17, 2026 08:20
b108db5 to
2eafbd8
Compare
…mand Add a CLI command to reset the statistics counters of a single server: clear counters server <backend>/<server> [force] Motivation: 'clear counters all' resets counters across every proxy and server in the process, which is too blunt for common operational needs. In particular, when a server slot is being reused to represent a different logical entity (e.g. a different Kubernetes pod occupying the same slot after 'set server <b>/<s> name'), the operator needs per-slot counter attribution and cannot afford to wipe the entire process' stats. The command requires ACCESS_LVL_ADMIN, like 'clear counters all'. It resets cumulative counters (not just the max gauges cleared by the base 'clear counters'), so restricting it to admin prevents an operator-level socket from hiding accumulated activity. Like 'clear counters' / 'clear counters all', it is not gated by the server's administrative state: the reset only zeroes counter values and does not touch the server object or its runtime state, so it is safe to issue on a live server (a concurrent counter increment races on a value exactly as it already does for 'clear counters all'). When the server's counters are registered in a shared-memory stats file object (COUNTERS_SHARED_F_LOCAL not set), clearing them breaks the monotonicity that monitoring tools consuming the shared stats rely on and affects every process attached to the object. Such a clear is therefore refused unless the optional 'force' argument is given. The reset covers both the native server counters, via counters_be_reset() introduced earlier in this series (which safely zeroes the per-tgroup accumulated counter contents while preserving the shared.tg pointer array), and the module-registered extra counters, via the new srv_stats_clear_extra_counters() helper. Note on the dispatch: 'clear counters' is a two-word CLI keyword, and cli_find_kw() matches the first keyword whose tokens are all consumed, so a separate three-word 'clear counters server' keyword would be shadowed (or would shadow) depending on registration order. To avoid that ambiguity the new sub-command is dispatched from within cli_parse_clear_counters() when args[2] == "server", mirroring how the existing "all" argument is handled. The per-server worker lives in server.c (cli_clear_counters_server()) where the server lookup and lock helpers are available.
Cover the new 'clear counters server <backend>/<server>' CLI command:
- error cases:
* unknown backend
* unknown server
- success case: drive two requests pinned to a single server (the
second server is a backup, so it stays idle), verify via 'show stat'
that the served server accumulated sessions while the backup stayed
at zero, clear only the served server's counters, then verify its
cumulative sessions dropped back to zero while the backup is still
untouched.
- the optional 'force' argument is accepted (a no-op override here,
since these process-local counters are not backed by a shared-memory
stats file) and the command is discoverable via 'help'.
The success path also implicitly exercises the counters_be_reset()
helper: if the reset were unsafe against the shared.tg pointer, the
vtest run would segfault on the next scheduled scrape or health check.
Add a section for the new 'clear counters server <backend>/<server> [force]' CLI command in doc/management.txt, alongside 'clear counters' and 'clear counters all'. The entry documents the admin-only permission level (it clears accumulated counters, like 'clear counters all') and the fact that both accumulated counters and max values are reset, along with the clearable module-registered extra counters, while the server's runtime state (address, weight, admin state) and health checks are left untouched. It describes the optional 'force' argument required to clear counters stored in a shared-memory stats file, and calls out the Kubernetes slot-recycle use case that motivated the command (per-entity attribution after 'set server <b>/<s> name').
alexanderstephan
force-pushed
the
clear-counters-server
branch
from
July 17, 2026 08:30
2eafbd8 to
c555fbe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.