Skip to content
How many queries per second can be handled by zabbix-agent per defined item key?
Python
Branch: master
Clone or download
Latest commit 20b7ec7 Aug 9, 2017
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
README.md Update README.md Aug 9, 2017
zabbix-agent-stress-test.py Help added Feb 23, 2015
zabbix_agentd.conf First release Feb 22, 2015

README.md

Zabbix Agent Stress Test

Script for Zabbix Agent stress testing - how many queries per second can be reached for defined item key from zabbix-agent in passive mode?

Paypal donate button

Zabbix Agent performance depends on:

  • zabbix-agent config: how many passive threads are started - config option StartAgents
  • item, items can be slower, if they need subshell or IOPs operation, e.g. UserParameters
  • HW (CPU, network, ...)

Stress test code can be more precise and also async => provided results are only informative. Stress test is only for zabbix-agent in passive mode and maybe active mode can provide better performance (IDNK).

Manual

$ ./zabbix-agent-stress-test.py -h
Usage:
./zabbix-agent-stress-test.py [-h] [-s <host name or IP>] [-p <port>] -k <key>

Utility for stress testing of zabbix_agent - how many queries per second can be reached for defined item key.

Options:
  -s, --host <host name or IP>
    Specify host name or IP address of a host. Default value is 127.0.0.1

  -p, --port <port>
    Specify port number of agent running on the host. Default value is 10050

  -k, --key <key of metric>
    Specify key of item to retrieve value for

  -t, --threads <number of thread>
    Specify number of worker threads

  -h, --help
    Display help information

Example: ./zabbix-agent-stress-test.py -s 127.0.0.1 -p 10050 -k agent.ping

Stress test examples

Some examples for Zabbix agent 2.4.3 on localhost and StartAgents=4:

Expected ~4 qps, because 4 agents threads are started and every execution needs 1 sec (sleep 1):

$ ./zabbix-agent-stress-test.py -s 127.0.0.1 -k "system.run[sleep 1]" -t 20
Warning: you are starting more threads, than your system has available CPU cores (2)!
Starting 20 threads, host: 127.0.0.1:10050, key: system.run[sleep 1]
Success: 4      Errors: 0       Avg rate: 18.55 qps    Execution time: 1.00 sec
Success: 7      Errors: 0       Avg rate: 11.04 qps    Execution time: 2.00 sec
Success: 11     Errors: 0       Avg rate: 7.13 qps     Execution time: 3.00 sec
Success: 12     Errors: 0       Avg rate: 6.88 qps     Execution time: 4.00 sec
Success: 16     Errors: 0       Avg rate: 5.10 qps     Execution time: 5.01 sec
Success: 20     Errors: 0       Avg rate: 4.05 qps     Execution time: 6.01 sec
Success: 24     Errors: 0       Avg rate: 3.98 qps     Execution time: 7.01 sec
Success: 28     Errors: 0       Avg rate: 3.97 qps     Execution time: 8.01 sec
Success: 32     Errors: 0       Avg rate: 3.96 qps     Execution time: 9.01 sec
Success: 36     Errors: 0       Avg rate: 3.96 qps     Execution time: 10.02 sec
Success: 40     Errors: 0       Avg rate: 3.96 qps     Execution time: 11.02 sec
Success: 44     Errors: 0       Avg rate: 3.96 qps     Execution time: 12.02 sec
Success: 48     Errors: 0       Avg rate: 3.96 qps     Execution time: 13.02 sec
Success: 52     Errors: 0       Avg rate: 3.97 qps     Execution time: 14.03 sec
Success: 56     Errors: 0       Avg rate: 3.98 qps     Execution time: 15.03 sec
...

Expected ~400 qps value, because 4 agents threads are started and execution needs ~0.01 sec (echo 1):

$ ./zabbix-agent-stress-test.py -s 127.0.0.1 -k "system.run[echo 1]" -t 20
Warning: you are starting more threads, than your system has available CPU cores (2)!
Starting 20 threads, host: 127.0.0.1:10050, key: system.run[echo 1]
Success: 596    Errors: 0       Avg rate: 525.18 qps   Execution time: 1.00 sec
Success: 1144   Errors: 0       Avg rate: 564.76 qps   Execution time: 2.00 sec
Success: 1673   Errors: 0       Avg rate: 479.72 qps   Execution time: 3.00 sec
Success: 2230   Errors: 0       Avg rate: 646.48 qps   Execution time: 4.00 sec
Success: 2808   Errors: 0       Avg rate: 577.59 qps   Execution time: 5.01 sec
Success: 3357   Errors: 0       Avg rate: 532.59 qps   Execution time: 6.01 sec
Success: 3950   Errors: 0       Avg rate: 589.85 qps   Execution time: 7.01 sec
Success: 4536   Errors: 0       Avg rate: 527.77 qps   Execution time: 8.01 sec
Success: 5112   Errors: 0       Avg rate: 595.04 qps   Execution time: 9.01 sec
Success: 5686   Errors: 0       Avg rate: 620.66 qps   Execution time: 10.01 sec
Success: 6247   Errors: 0       Avg rate: 600.07 qps   Execution time: 11.01 sec
Success: 6802   Errors: 0       Avg rate: 521.53 qps   Execution time: 12.01 sec
Success: 7362   Errors: 0       Avg rate: 548.17 qps   Execution time: 13.01 sec
Success: 7933   Errors: 0       Avg rate: 580.31 qps   Execution time: 14.01 sec
...

Probably maximum qps value, when 4 agents threads are started - item key is agent.ping, so no subshell executions or IOPs are needed for this item:

$ ./zabbix-agent-stress-test.py -s 127.0.0.1 -k "agent.ping" -t 4
Warning: you are starting more threads, than your system has available CPU cores (2)!
Starting 4 threads, host: 127.0.0.1:10050, key: agent.ping
Success: 3354   Errors: 0       Avg rate: 3406.18 qps  Execution time: 1.00 sec
Success: 6692   Errors: 0       Avg rate: 4054.38 qps  Execution time: 2.00 sec
Success: 9952   Errors: 0       Avg rate: 3347.73 qps  Execution time: 3.00 sec
Success: 13395  Errors: 0       Avg rate: 3476.23 qps  Execution time: 4.00 sec
Success: 16511  Errors: 0       Avg rate: 3946.44 qps  Execution time: 5.00 sec
Success: 20041  Errors: 0       Avg rate: 4049.99 qps  Execution time: 6.01 sec
Success: 23502  Errors: 0       Avg rate: 8685.35 qps  Execution time: 7.01 sec
Success: 26875  Errors: 0       Avg rate: 5739.68 qps  Execution time: 8.01 sec
Success: 30107  Errors: 0       Avg rate: 6029.05 qps  Execution time: 9.01 sec
Success: 33344  Errors: 0       Avg rate: 3814.14 qps  Execution time: 10.01 sec
...

Conclusion

Zabbix Agent can handle ~3k requests per second for in memory items. If you need shell execution for items, then it's ~0.5k requests per second. (Tested on 2.1GHz CPUs).

Similar projects

Better implementation in Go: https://github.com/cavaliercoder/zabbix_agent_bench

Author

Devops Monitoring Expert, who loves monitoring systems and cutting/bleeding edge technologies: Docker, Kubernetes, ECS, AWS, Google GCP, Terraform, Lambda, Zabbix, Grafana, Elasticsearch, Kibana, Prometheus, Sysdig, ...

Summary:

Professional devops / monitoring / consulting services:

Monitoring Artist

You can’t perform that action at this time.