HATop is an interactive ncurses client and real-time monitoring, statistics displaying tool for the HAProxy TCP/HTTP load balancer.
HATop's appearance is similar to top(1). It supports various modes for detailed statistics of all configured proxies and services in near realtime. In addition, it features an interactive CLI for the haproxy unix socket. This allows administrators to control the given haproxy instance (change server weight, put servers into maintenance mode, ...) directly out of hatop and monitor the results immediately.
It is important to understand that when multiple haproxy processes are started on the same socket, any process may pick up the request and thus hatop will output stats owned solely by that process. The current haproxy-internal process id is displayed top right.
See INSTALL
or refer to install
Invoking hatop without options or with -h / --help
results in:
$ hatop --help
Usage: hatop (-s SOCKET| -t HOST:PORT) [OPTIONS]...
Options:
--version show program's version number and exit
-h, --help show this help message and exit
Mandatory:
-s SOCKET, --unix-socket=SOCKET
path to the haproxy unix socket
-t TCP_SOCKET, --tcp-socket=TCP_SOCKET
address of the haproxy tcp stats socket
Optional:
-i INTERVAL, --update-interval=INTERVAL
update interval in seconds (1-30, default: 3)
-m MODE, --mode=MODE
start in specific mode (1-5, default: 1)
-n, --read-only disable the cli and query for stats only
Filters:
Note: All filter options may be given multiple times.
-f FILTER, --filter=FILTER
stat filter in format "<iid> <type> <sid>"
-p PROXY, --proxy=PROXY
proxy filter in format "<pxname>"
See also: screenshots
ID Mode Description
1 STATUS The default mode with health, session and queue statistics
2 TRAFFIC Display connection and request rates as well as traffic stats
3 HTTP Display various statistical information related to HTTP
4 ERRORS Display health info, various error counters and downtimes
5 CLI Display embedded command line client for the unix socket
See also: keybinds
Key Action
Hh? Display this help screen
CTRL-C / Qq Quit
TAB Cycle mode forwards
SHIFT-TAB Cycle mode backwards
ALT-n / ESC-n Switch to mode n, where n is the numeric mode id
ESC-ESC Jump to previous mode
ENTER Display hotkey menu for selected service
SPACE Copy and paste selected service identifier to the CLI
You can scroll the stat views using UP / DOWN / PGUP / PGDOWN / HOME / END
.
The reverse colored cursor line is used to select a given service instance.
An unique identifier [#<iid>/<#sid>]
of the selected service is displayed bottom right.
You can hit SPACE
to copy and paste the identifier in string format pxname/svname
to the CLI for easy re-use with some commands.
For example:
- Open the CLI
- Type "disable server "
- Switch back to some stat view using TAB / SHIFT-TAB
- Select the server instance using UP / DOWN
- Hit SPACE
The result is this command line:
> disable server <pxname>/<svname>
Hotkey Action
F1 Enable server on all backends (return from maintenance mode)
F2 Disable server on all backends (put into maintenance mode)
F3 Drain server on all backends (prepare for maintenance mode, don't allow new connections)
F4 Restore initial server weight
F5 Decrease server weight: - 10
F6 Decrease server weight: - 1
F7 Increase server weight: + 1
F8 Increase server weight: + 10
F9 Enable server on one backend (return from maintenance mode)
F10 Disable server on one backend (put into maintenance mode)
F11 Drain server on one backend (prepare for maintenance mode, don't allow new connections)
Hotkey actions and server responses are logged on the CLI viewport.
You can scroll the output on the CLI view using PGUP / PGDOWN
.
A brief keybind reference is logged there directly after startup...
See also: screenshots
Node configured name of the haproxy node
Uptime runtime since haproxy was initially started
Pipes pipes are currently used for kernel-based tcp slicing
Procs number of haproxy processes
Tasks number of actice process tasks
Queue number of queued process tasks (run queue)
Proxies number of configured proxies
Services number of configured services
NAME name of the proxy and its services
W configured weight of the service
STATUS service status (UP/DOWN/NOLB/MAINT/MAINT(via)...)
CHECK status of last health check (see status reference below)
ACT server is active (server), number of active servers (backend)
BCK server is backup (server), number of backup servers (backend)
QCUR current queued requests
QMAX max queued requests
SCUR current sessions
SMAX max sessions
SLIM sessions limit
STOT total sessions
LBTOT total number of times a server was selected
RATE number of sessions per second over last elapsed second
RLIM limit on new sessions per second
RMAX max number of new sessions per second
BIN bytes in (IEEE 1541-2002)
BOUT bytes out (IEEE 1541-2002)
RATE HTTP requests per second over last elapsed second
RMAX max number of HTTP requests per second observed
RTOT total number of HTTP requests received
1xx number of HTTP responses with 1xx code
2xx number of HTTP responses with 2xx code
3xx number of HTTP responses with 3xx code
4xx number of HTTP responses with 4xx code
5xx number of HTTP responses with 5xx code
?xx number of HTTP responses with other codes (protocol error)
CF number of failed checks
CD number of UP->DOWN transitions
CL last status change
ECONN connection errors
EREQ request errors
ERSP response errors
DREQ denied requests
DRSP denied responses
DOWN total downtime
UNK unknown
INI initializing
SOCKERR socket error
L4OK check passed on layer 4, no upper layers testing enabled
L4TMOUT layer 1-4 timeout
L4CON layer 1-4 connection problem, for example
"Connection refused" (tcp rst) or "No route to host" (icmp)
L6OK check passed on layer 6
L6TOUT layer 6 (SSL) timeout
L6RSP layer 6 invalid response - protocol error
L7OK check passed on layer 7
L7OKC check conditionally passed on layer 7, for example 404 with
disable-on-404
L7TOUT layer 7 (HTTP/SMTP) timeout
L7RSP layer 7 invalid response - protocol error
L7STS layer 7 response error, for example HTTP 5xx
HATop was originally written by John Feuerstein <john@feurix.com>, known on GitHub as @feurix.
As of 2020, however, support for Python 2.x is being phased out of modern distributions of Linux, the last code interaction on the upstream repository (<https://github.com/feurix/hatop>) is from over four years ago, and no issue or pull request has been able to elicit a response from the author.
As of February 29th (2020), I have forked this repository into my own GitHub org, at <https://github.com/jhunt/hatop>, with the intent of merging some of the upstream pull requests, and continuing ongoing maintenance of this tool.
I have been an avid user of HATop since I first discovered it many many years ago, and would like to continue to be able to use it for many many more.
The following people have contributed to HATop:
- Andrew Hayworth
- Cyril Bonté
- James Hunt
- John Feuerstein
- Louis Charreau
- Matt Behrens