Slower wazuh-db in 4.8.0 #23347
Assignees
Labels
Comments
Closed
4 tasks
4 tasks
UpdateI followed the steps to reproduce it and got these results: v4.7.4root@vm-ubuntu2204-server:/var/ossec/etc# tail -f /var/ossec/logs/cluster.log | grep -i agent-info
2024/05/09 21:19:21 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:19:21 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.474s. Updated 166 chunks.
2024/05/09 21:19:32 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:19:32 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.963s. Updated 166 chunks.
2024/05/09 21:19:42 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:19:43 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.548s. Updated 166 chunks.
2024/05/09 21:19:52 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:19:53 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.487s. Updated 166 chunks.
2024/05/09 21:20:03 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:20:04 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.483s. Updated 166 chunks.
2024/05/09 21:20:14 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:20:14 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.460s. Updated 166 chunks.
2024/05/09 21:20:24 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:20:25 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.507s. Updated 166 chunks.
2024/05/09 21:20:35 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:20:35 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.692s. Updated 166 chunks.
2024/05/09 21:20:45 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:20:46 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.436s. Updated 166 chunks.
2024/05/09 21:20:56 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:20:56 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.417s. Updated 166 chunks.
2024/05/09 21:21:06 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:21:07 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.420s. Updated 166 chunks.
2024/05/09 21:21:17 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:21:17 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.489s. Updated 166 chunks.
2024/05/09 21:21:27 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:21:28 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.500s. Updated 166 chunks.
2024/05/09 21:21:38 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:21:39 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.723s. Updated 166 chunks.
2024/05/09 21:21:48 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:21:49 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.483s. Updated 166 chunks.
2024/05/09 21:21:59 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:21:59 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.463s. Updated 166 chunks.
2024/05/09 21:22:09 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:22:10 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.500s. Updated 166 chunks.
2024/05/09 21:22:20 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:22:20 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.477s. Updated 166 chunks.
2024/05/09 21:22:30 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:22:31 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.519s. Updated 166 chunks.
2024/05/09 21:22:41 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 21:22:42 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.646s. Updated 166 chunks.Average: 0.534s v4.8.0root@vm-ubuntu2204-server:/vagrant# tail -f /var/ossec/logs/cluster.log | grep -i agent-info
2024/05/09 18:56:08 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:56:08 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.440s. Updated 166 chunks.
2024/05/09 18:56:18 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:56:19 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.451s. Updated 166 chunks.
2024/05/09 18:56:29 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:56:30 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.716s. Updated 166 chunks.
2024/05/09 18:56:39 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:56:40 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.504s. Updated 166 chunks.
2024/05/09 18:56:50 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:56:50 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.457s. Updated 166 chunks.
2024/05/09 18:57:00 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:57:01 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.434s. Updated 166 chunks.
2024/05/09 18:57:11 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:57:11 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.547s. Updated 166 chunks.
2024/05/09 18:57:21 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:57:22 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.475s. Updated 166 chunks.
2024/05/09 18:57:32 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:57:33 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.712s. Updated 166 chunks.
2024/05/09 18:57:43 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:57:43 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.505s. Updated 166 chunks.
2024/05/09 18:57:53 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:57:53 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.428s. Updated 166 chunks.
2024/05/09 18:58:04 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:58:04 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.497s. Updated 166 chunks.
2024/05/09 18:58:14 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:58:15 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.426s. Updated 166 chunks.
2024/05/09 18:58:25 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:58:25 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.419s. Updated 166 chunks.
2024/05/09 18:58:35 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:58:36 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.739s. Updated 166 chunks.
2024/05/09 18:58:46 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:58:46 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.507s. Updated 166 chunks.
2024/05/09 18:58:56 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:58:57 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.452s. Updated 166 chunks.
2024/05/09 18:59:07 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:59:07 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.475s. Updated 166 chunks.
2024/05/09 18:59:17 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:59:18 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.443s. Updated 166 chunks.
2024/05/09 18:59:28 INFO: [Worker wazuh-2] [Agent-info sync] Starting.
2024/05/09 18:59:29 INFO: [Worker wazuh-2] [Agent-info sync] Finished in 0.400s. Updated 166 chunks.Average: 0.501s No increase in sync times is observed for v4.8.0, in fact the average is minimally lower than for v4.7.4. |
|
We have analyzed with @Selutario the results of the test I did locally and apparently it is fine. In the original test it failed only 1 out of 25 workers, it is probable that this will not happen in the local test. So we have decided to re-launch the 4.8.0-rc1 build: https://ci.wazuh.info/job/CLUSTER-Workload_benchmarks_metrics/517/. Once finished, we observed with @cborla and @nico-stefani that it had not been executed with the cluster performance test parameters. So we have had to relaunch it again including these parameters: https://ci.wazuh.info/job/CLUSTER-Workload_benchmarks_metrics/518/. We are waiting for it to be completed to analyze the results. |
@nbertoldo, @cborla Luckily, it is possible to run the performance test on the artifacts generated by the pipeline. I am attaching them here. Build 517 (v4.8.0-rc1)Artifacts: artifacts_480_rc1_B517.zip
It is also possible to break down the statistics used in the test: Tasks stats>>> from wazuh_testing.tools.performance.csv_parser import ClusterCSVTasksParser
>>> ClusterCSVTasksParser('/home/selu/Descargas/cluster_performance/517/artifacts_480_rc1').get_stats()
{
"setup_phase": {
"integrity_check": {
"time_spent(s)": {
"workers": {
"mean": ("worker_17", 0.3481111111111111),
"max": ("worker_14", 3.176),
},
"master": {
"mean": ("master", 0.05240245824141191),
"max": ("master", 0.709),
},
}
},
"integrity_sync": {
"time_spent(s)": {
"workers": {
"mean": ("worker_8", 0.04211764705882353),
"max": ("worker_23", 0.163),
},
"master": {
"mean": ("master", 0.5421203007518796),
"max": ("master", 3.217),
},
}
},
"agent-info_sync": {
"time_spent(s)": {
"workers": {
"mean": ("worker_18", 0.9509827586206897),
"max": ("worker_9", 10.639),
},
"master": {
"mean": ("master", 0.687005693950178),
"max": ("master", 10.257),
},
}
},
},
"stable_phase": {
"integrity_check": {
"time_spent(s)": {
"workers": {
"mean": ("worker_3", 0.01140740740740741),
"max": ("worker_3", 0.04),
},
"master": {
"mean": ("master", 0.00456888888888889),
"max": ("master", 0.017),
},
}
},
"agent-info_sync": {
"time_spent(s)": {
"workers": {"mean": ("worker_18", 0.00964), "max": ("worker_18", 0.025)}
}
},
},
}Resources stats>>> from wazuh_testing.tools.performance.csv_parser import ClusterCSVResourcesParser
>>> ClusterCSVResourcesParser('/home/selu/Descargas/cluster_performance/517/artifacts_480_rc1').get_stats()
{
"setup_phase": {
"wazuh-clusterd": {
"USS(KB)": {
"workers": {
"mean": ("worker_14", 60914.15625),
"max": ("worker_14", 80292.0),
"reg_cof": ("worker_7", 169.06954404578758),
},
"master": {
"mean": ("master", 66037.34439834025),
"max": ("master", 103936.0),
"reg_cof": ("master", 170.65657213401462),
},
},
"CPU(%)": {
"workers": {
"mean": ("worker_17", 1.51796875),
"max": ("worker_5", 5.0),
"reg_cof": ("worker_21", -0.004918877477017017),
},
"master": {
"mean": ("master", 36.231950207468884),
"max": ("master", 170.0),
"reg_cof": ("master", -0.18016486060148834),
},
},
"FD": {
"workers": {
"mean": ("worker_18", 73.79296875),
"max": ("worker_14", 77),
"reg_cof": ("worker_3", 0.01055827858832108),
},
"master": {
"mean": ("master", 133.09958506224066),
"max": ("master", 141.0),
"reg_cof": ("master", -0.019704399711944067),
},
},
}
},
"stable_phase": {
"wazuh-clusterd": {
"USS(KB)": {
"workers": {
"mean": ("worker_21", 72323.91111111111),
"max": ("worker_21", 72324.0),
"reg_cof": ("worker_3", 1.0142353770271475),
}
},
"CPU(%)": {
"workers": {
"mean": ("worker_4", 0.19999999999999998),
"max": ("worker_20", 3.0),
"reg_cof": ("worker_12", 0.0009584214235377039),
}
},
"FD": {
"workers": {
"mean": ("worker_4", 75.0),
"max": ("worker_4", 75),
"reg_cof": ("worker_4", 1.6472884900208011e-15),
}
},
}
},
}I was also executing requests against the stats endpoint to obtain statistics from wazuh-db and other services, both from the master and from a random worker (21). However, since everything worked fine in this build, they don't provide very useful information: Services statsBefore restarting ( Mastercurl -k -X GET "https://localhost:55000/cluster/master/daemons/stats?pretty=true" -H "Authorization: Bearer $TOKEN"
{
"data": {
"affected_items": [
{
"uptime": "2024-05-10T15:13:59+00:00",
"timestamp": "2024-05-10T15:38:09+00:00",
"name": "wazuh-remoted",
"metrics": {
"bytes": {
"received": 0,
"sent": 0
},
"keys_reload_count": 47,
"messages": {
"received_breakdown": {
"control": 0,
"control_breakdown": {
"keepalive": 0,
"request": 0,
"shutdown": 0,
"startup": 0
},
"dequeued_after": 0,
"discarded": 0,
"event": 0,
"ping": 0,
"unknown": 0
},
"sent_breakdown": {
"ack": 0,
"ar": 0,
"discarded": 0,
"request": 0,
"sca": 0,
"shared": 0
}
},
"queues": {
"received": {
"size": 131072,
"usage": 0
}
},
"tcp_sessions": 0
}
},
{
"uptime": "2024-05-10T15:14:00+00:00",
"timestamp": "2024-05-10T15:38:09+00:00",
"name": "wazuh-analysisd",
"metrics": {
"bytes": {
"received": 580524
},
"events": {
"processed": 1365,
"received": 2346,
"received_breakdown": {
"decoded_breakdown": {
"agent": 0,
"agentless": 0,
"dbsync": 979,
"integrations_breakdown": {
"virustotal": 0
},
"modules_breakdown": {
"aws": 0,
"azure": 0,
"ciscat": 0,
"command": 0,
"docker": 0,
"gcp": 0,
"github": 0,
"logcollector_breakdown": {
"eventchannel": 0,
"eventlog": 0,
"macos": 0,
"others": 1360
},
"office365": 0,
"ms-graph": 0,
"oscap": 0,
"osquery": 0,
"rootcheck": 2,
"sca": 2,
"syscheck": 2,
"syscollector": 0,
"upgrade": 0,
"vulnerability": 0
},
"monitor": 1,
"remote": 0,
"syslog": 0
},
"dropped_breakdown": {
"agent": 0,
"agentless": 0,
"dbsync": 0,
"integrations_breakdown": {
"virustotal": 0
},
"modules_breakdown": {
"aws": 0,
"azure": 0,
"ciscat": 0,
"command": 0,
"docker": 0,
"gcp": 0,
"github": 0,
"logcollector_breakdown": {
"eventchannel": 0,
"eventlog": 0,
"macos": 0,
"others": 0
},
"office365": 0,
"ms-graph": 0,
"oscap": 0,
"osquery": 0,
"rootcheck": 0,
"sca": 0,
"syscheck": 0,
"syscollector": 0,
"upgrade": 0,
"vulnerability": 0
},
"monitor": 0,
"remote": 0,
"syslog": 0
}
},
"written_breakdown": {
"alerts": 82,
"archives": 0,
"firewall": 0,
"fts": 0,
"stats": 0
}
},
"queues": {
"alerts": {
"size": 16384,
"usage": 0
},
"archives": {
"size": 16384,
"usage": 0
},
"dbsync": {
"size": 16384,
"usage": 0
},
"eventchannel": {
"size": 16384,
"usage": 0
},
"firewall": {
"size": 16384,
"usage": 0
},
"fts": {
"size": 16384,
"usage": 0
},
"hostinfo": {
"size": 16384,
"usage": 0
},
"others": {
"size": 16384,
"usage": 0
},
"processed": {
"size": 16384,
"usage": 0
},
"rootcheck": {
"size": 16384,
"usage": 0
},
"sca": {
"size": 16384,
"usage": 0
},
"stats": {
"size": 16384,
"usage": 0
},
"syscheck": {
"size": 16384,
"usage": 0
},
"syscollector": {
"size": 16384,
"usage": 0
},
"upgrade": {
"size": 16384,
"usage": 0
}
}
}
},
{
"uptime": "2024-05-10T15:13:58+00:00",
"timestamp": "2024-05-10T15:38:09+00:00",
"name": "wazuh-db",
"metrics": {
"queries": {
"received": 297596,
"received_breakdown": {
"agent": 989,
"agent_breakdown": {
"db": {
"begin": 0,
"close": 0,
"commit": 0,
"remove": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 2
},
"sca": {
"sca": 6
},
"sync": {
"dbsync": 0
},
"syscheck": {
"fim_file": 64,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 2
},
"syscollector": {
"syscollector_hotfixes": 1,
"syscollector_hwinfo": 2,
"syscollector_network_address": 1,
"syscollector_network_iface": 2,
"syscollector_network_protocol": 1,
"syscollector_osinfo": 2,
"syscollector_packages": 626,
"syscollector_ports": 20,
"syscollector_processes": 260,
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
}
}
}
},
"global": 294874,
"global_breakdown": {
"db": {
"backup": 0,
"sql": 473,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"agent": {
"delete-agent": 0,
"disconnect-agents": 2,
"find-agent": 0,
"get-agent-info": 150003,
"get-agents-by-connection-status": 0,
"get-all-agents": 1,
"get-distinct-groups": 145,
"get-groups-integrity": 0,
"insert-agent": 50003,
"reset-agents-connection": 1,
"select-agent-group": 0,
"select-agent-name": 1,
"set-agent-groups": 79152,
"sync-agent-groups-get": 297,
"sync-agent-info-get": 0,
"sync-agent-info-set": 14788,
"update-agent-data": 1,
"update-agent-name": 1,
"update-connection-status": 0,
"update-status-code": 0,
"update-keepalive": 0
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 2,
"find-group": 2,
"insert-agent-group": 1,
"select-groups": 1
},
"labels": {
"get-labels": 0
}
}
},
"mitre": 1702,
"mitre_breakdown": {
"db": {
"sql": 1702
}
},
"task": 31,
"task_breakdown": {
"db": {
"sql": 2
},
"tables": {
"tasks": {
"delete_old": 1,
"set_timeout": 2,
"upgrade": 0,
"upgrade_cancel_tasks": 26,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 0,
"wazuhdb_breakdown": {
"db": {
"remove": 0
}
}
}
},
"time": {
"execution": 1586000,
"execution_breakdown": {
"agent": 228,
"agent_breakdown": {
"db": {
"open": 38,
"begin": 0,
"close": 0,
"commit": 0,
"remove": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 3
},
"sca": {
"sca": 0
},
"sync": {
"dbsync": 0
},
"syscheck": {
"fim_file": 15,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 41
},
"syscollector": {
"syscollector_hotfixes": 0,
"syscollector_hwinfo": 0,
"syscollector_network_address": 0,
"syscollector_network_iface": 0,
"syscollector_network_protocol": 0,
"syscollector_osinfo": 0,
"syscollector_packages": 108,
"syscollector_ports": 1,
"syscollector_processes": 17,
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
}
}
}
},
"global": 1584778,
"global_breakdown": {
"db": {
"open": 869160,
"backup": 0,
"sql": 3755,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"agent": {
"delete-agent": 0,
"disconnect-agents": 23,
"find-agent": 0,
"get-agent-info": 3858,
"get-agents-by-connection-status": 0,
"get-all-agents": 0,
"get-distinct-groups": 1416,
"get-groups-integrity": 0,
"insert-agent": 3484,
"reset-agents-connection": 0,
"select-agent-group": 0,
"select-agent-name": 0,
"set-agent-groups": 624287,
"sync-agent-groups-get": 6285,
"sync-agent-info-get": 0,
"sync-agent-info-set": 72447,
"update-agent-data": 0,
"update-agent-name": 1,
"update-connection-status": 0,
"update-status-code": 0,
"update-keepalive": 0
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 56,
"find-group": 0,
"insert-agent-group": 0,
"select-groups": 0
},
"labels": {
"get-labels": 0
}
}
},
"mitre": 990,
"mitre_breakdown": {
"db": {
"sql": 990
}
},
"task": 4,
"task_breakdown": {
"db": {
"sql": 0
},
"tables": {
"tasks": {
"delete_old": 1,
"set_timeout": 2,
"upgrade": 0,
"upgrade_cancel_tasks": 1,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 0,
"wazuhdb_breakdown": {
"db": {
"remove": 0
}
}
}
}
}
}
],
"total_affected_items": 3,
"total_failed_items": 0,
"failed_items": []
},
"message": "Statistical information for each daemon was successfully read",
"error": 0
}Worker 21curl -k -X GET "https://localhost:55000/cluster/CLUSTER-Workload_benchmarks_metrics_B517_manager_21/daemons/stats?pretty=true" -H "Authorization: Bearer $TOKEN"
{
"data": {
"affected_items": [
{
"uptime": "2024-05-10T15:14:01+00:00",
"timestamp": "2024-05-10T15:38:09+00:00",
"name": "wazuh-remoted",
"metrics": {
"bytes": {
"received": 46394200,
"sent": 2096757528
},
"keys_reload_count": 38,
"messages": {
"received_breakdown": {
"control": 59188,
"control_breakdown": {
"keepalive": 57349,
"request": 0,
"shutdown": 0,
"startup": 1843
},
"dequeued_after": 0,
"discarded": 0,
"event": 0,
"ping": 0,
"unknown": 0
},
"sent_breakdown": {
"ack": 59192,
"ar": 5335,
"discarded": 1,
"request": 0,
"sca": 0,
"shared": 5303775
}
},
"queues": {
"received": {
"size": 131072,
"usage": 112730
}
},
"tcp_sessions": 1990
}
},
{
"uptime": "2024-05-10T15:14:02+00:00",
"timestamp": "2024-05-10T15:38:09+00:00",
"name": "wazuh-analysisd",
"metrics": {
"bytes": {
"received": 525345
},
"events": {
"processed": 1200,
"received": 2145,
"received_breakdown": {
"decoded_breakdown": {
"agent": 0,
"agentless": 0,
"dbsync": 943,
"integrations_breakdown": {
"virustotal": 0
},
"modules_breakdown": {
"aws": 0,
"azure": 0,
"ciscat": 0,
"command": 0,
"docker": 0,
"gcp": 0,
"github": 0,
"logcollector_breakdown": {
"eventchannel": 0,
"eventlog": 0,
"macos": 0,
"others": 1195
},
"office365": 0,
"ms-graph": 0,
"oscap": 0,
"osquery": 0,
"rootcheck": 2,
"sca": 2,
"syscheck": 2,
"syscollector": 0,
"upgrade": 0,
"vulnerability": 0
},
"monitor": 1,
"remote": 0,
"syslog": 0
},
"dropped_breakdown": {
"agent": 0,
"agentless": 0,
"dbsync": 0,
"integrations_breakdown": {
"virustotal": 0
},
"modules_breakdown": {
"aws": 0,
"azure": 0,
"ciscat": 0,
"command": 0,
"docker": 0,
"gcp": 0,
"github": 0,
"logcollector_breakdown": {
"eventchannel": 0,
"eventlog": 0,
"macos": 0,
"others": 0
},
"office365": 0,
"ms-graph": 0,
"oscap": 0,
"osquery": 0,
"rootcheck": 0,
"sca": 0,
"syscheck": 0,
"syscollector": 0,
"upgrade": 0,
"vulnerability": 0
},
"monitor": 0,
"remote": 0,
"syslog": 0
}
},
"written_breakdown": {
"alerts": 58,
"archives": 0,
"firewall": 0,
"fts": 0,
"stats": 0
}
},
"queues": {
"alerts": {
"size": 16384,
"usage": 0
},
"archives": {
"size": 16384,
"usage": 0
},
"dbsync": {
"size": 16384,
"usage": 0
},
"eventchannel": {
"size": 16384,
"usage": 0
},
"firewall": {
"size": 16384,
"usage": 0
},
"fts": {
"size": 16384,
"usage": 0
},
"hostinfo": {
"size": 16384,
"usage": 0
},
"others": {
"size": 16384,
"usage": 0
},
"processed": {
"size": 16384,
"usage": 0
},
"rootcheck": {
"size": 16384,
"usage": 0
},
"sca": {
"size": 16384,
"usage": 0
},
"stats": {
"size": 16384,
"usage": 0
},
"syscheck": {
"size": 16384,
"usage": 0
},
"syscollector": {
"size": 16384,
"usage": 0
},
"upgrade": {
"size": 16384,
"usage": 0
}
}
}
},
{
"uptime": "2024-05-10T15:14:00+00:00",
"timestamp": "2024-05-10T15:38:09+00:00",
"name": "wazuh-db",
"metrics": {
"queries": {
"received": 149324,
"received_breakdown": {
"agent": 953,
"agent_breakdown": {
"db": {
"begin": 0,
"close": 0,
"commit": 0,
"remove": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 2
},
"sca": {
"sca": 6
},
"sync": {
"dbsync": 0
},
"syscheck": {
"fim_file": 41,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 2
},
"syscollector": {
"syscollector_hotfixes": 1,
"syscollector_hwinfo": 2,
"syscollector_network_address": 1,
"syscollector_network_iface": 2,
"syscollector_network_protocol": 1,
"syscollector_osinfo": 2,
"syscollector_packages": 626,
"syscollector_ports": 20,
"syscollector_processes": 247,
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
}
}
}
},
"global": 146971,
"global_breakdown": {
"db": {
"backup": 0,
"sql": 10,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"agent": {
"delete-agent": 0,
"disconnect-agents": 2,
"find-agent": 0,
"get-agent-info": 0,
"get-agents-by-connection-status": 223,
"get-all-agents": 236,
"get-distinct-groups": 145,
"get-groups-integrity": 0,
"insert-agent": 50000,
"reset-agents-connection": 1,
"select-agent-group": 36201,
"select-agent-name": 1,
"set-agent-groups": 141,
"sync-agent-groups-get": 167,
"sync-agent-info-get": 649,
"sync-agent-info-set": 0,
"update-agent-data": 36201,
"update-agent-name": 1,
"update-connection-status": 0,
"update-status-code": 0,
"update-keepalive": 22988
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 1,
"find-group": 2,
"insert-agent-group": 1,
"select-groups": 1
},
"labels": {
"get-labels": 0
}
}
},
"mitre": 1400,
"mitre_breakdown": {
"db": {
"sql": 1400
}
},
"task": 0,
"task_breakdown": {
"db": {
"sql": 0
},
"tables": {
"tasks": {
"delete_old": 0,
"set_timeout": 0,
"upgrade": 0,
"upgrade_cancel_tasks": 0,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 0,
"wazuhdb_breakdown": {
"db": {
"remove": 0
}
}
}
},
"time": {
"execution": 39937,
"execution_breakdown": {
"agent": 1608,
"agent_breakdown": {
"db": {
"open": 917,
"begin": 0,
"close": 0,
"commit": 0,
"remove": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 6
},
"sca": {
"sca": 2
},
"sync": {
"dbsync": 0
},
"syscheck": {
"fim_file": 47,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 4
},
"syscollector": {
"syscollector_hotfixes": 0,
"syscollector_hwinfo": 0,
"syscollector_network_address": 0,
"syscollector_network_iface": 2,
"syscollector_network_protocol": 1,
"syscollector_osinfo": 3,
"syscollector_packages": 599,
"syscollector_ports": 1,
"syscollector_processes": 22,
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
}
}
}
},
"global": 38215,
"global_breakdown": {
"db": {
"open": 5230,
"backup": 0,
"sql": 36,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"agent": {
"delete-agent": 0,
"disconnect-agents": 15,
"find-agent": 0,
"get-agent-info": 0,
"get-agents-by-connection-status": 2596,
"get-all-agents": 12699,
"get-distinct-groups": 864,
"get-groups-integrity": 0,
"insert-agent": 1173,
"reset-agents-connection": 0,
"select-agent-group": 793,
"select-agent-name": 0,
"set-agent-groups": 7186,
"sync-agent-groups-get": 1369,
"sync-agent-info-get": 2276,
"sync-agent-info-set": 0,
"update-agent-data": 2946,
"update-agent-name": 0,
"update-connection-status": 0,
"update-status-code": 0,
"update-keepalive": 991
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 34,
"find-group": 0,
"insert-agent-group": 0,
"select-groups": 0
},
"labels": {
"get-labels": 0
}
}
},
"mitre": 114,
"mitre_breakdown": {
"db": {
"sql": 114
}
},
"task": 0,
"task_breakdown": {
"db": {
"sql": 0
},
"tables": {
"tasks": {
"delete_old": 0,
"set_timeout": 0,
"upgrade": 0,
"upgrade_cancel_tasks": 0,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 0,
"wazuhdb_breakdown": {
"db": {
"remove": 0
}
}
}
}
}
}
],
"total_affected_items": 3,
"total_failed_items": 0,
"failed_items": []
},
"message": "Statistical information for each daemon was successfully read",
"error": 0
}After restarting the master node and before destroying the hosts (15:46:40): Mastercurl -k -X GET "https://localhost:55000/cluster/master/daemons/stats?pretty=true" -H "Authorization: Bearer $TOKEN"
{
"data": {
"affected_items": [
{
"uptime": "2024-05-10T15:39:47+00:00",
"timestamp": "2024-05-10T15:46:40+00:00",
"name": "wazuh-remoted",
"metrics": {
"bytes": {
"received": 0,
"sent": 0
},
"keys_reload_count": 0,
"messages": {
"received_breakdown": {
"control": 0,
"control_breakdown": {
"keepalive": 0,
"request": 0,
"shutdown": 0,
"startup": 0
},
"dequeued_after": 0,
"discarded": 0,
"event": 0,
"ping": 0,
"unknown": 0
},
"sent_breakdown": {
"ack": 0,
"ar": 0,
"discarded": 0,
"request": 0,
"sca": 0,
"shared": 0
}
},
"queues": {
"received": {
"size": 131072,
"usage": 0
}
},
"tcp_sessions": 0
}
},
{
"uptime": "2024-05-10T15:39:46+00:00",
"timestamp": "2024-05-10T15:46:40+00:00",
"name": "wazuh-analysisd",
"metrics": {
"bytes": {
"received": 222659
},
"events": {
"processed": 627,
"received": 839,
"received_breakdown": {
"decoded_breakdown": {
"agent": 0,
"agentless": 0,
"dbsync": 210,
"integrations_breakdown": {
"virustotal": 0
},
"modules_breakdown": {
"aws": 0,
"azure": 0,
"ciscat": 0,
"command": 0,
"docker": 0,
"gcp": 0,
"github": 0,
"logcollector_breakdown": {
"eventchannel": 0,
"eventlog": 0,
"macos": 0,
"others": 622
},
"office365": 0,
"ms-graph": 0,
"oscap": 0,
"osquery": 0,
"rootcheck": 2,
"sca": 2,
"syscheck": 2,
"syscollector": 0,
"upgrade": 0,
"vulnerability": 0
},
"monitor": 1,
"remote": 0,
"syslog": 0
},
"dropped_breakdown": {
"agent": 0,
"agentless": 0,
"dbsync": 0,
"integrations_breakdown": {
"virustotal": 0
},
"modules_breakdown": {
"aws": 0,
"azure": 0,
"ciscat": 0,
"command": 0,
"docker": 0,
"gcp": 0,
"github": 0,
"logcollector_breakdown": {
"eventchannel": 0,
"eventlog": 0,
"macos": 0,
"others": 0
},
"office365": 0,
"ms-graph": 0,
"oscap": 0,
"osquery": 0,
"rootcheck": 0,
"sca": 0,
"syscheck": 0,
"syscollector": 0,
"upgrade": 0,
"vulnerability": 0
},
"monitor": 0,
"remote": 0,
"syslog": 0
}
},
"written_breakdown": {
"alerts": 62,
"archives": 0,
"firewall": 0,
"fts": 0,
"stats": 0
}
},
"queues": {
"alerts": {
"size": 16384,
"usage": 0
},
"archives": {
"size": 16384,
"usage": 0
},
"dbsync": {
"size": 16384,
"usage": 0
},
"eventchannel": {
"size": 16384,
"usage": 0
},
"firewall": {
"size": 16384,
"usage": 0
},
"fts": {
"size": 16384,
"usage": 0
},
"hostinfo": {
"size": 16384,
"usage": 0
},
"others": {
"size": 16384,
"usage": 0
},
"processed": {
"size": 16384,
"usage": 0
},
"rootcheck": {
"size": 16384,
"usage": 0
},
"sca": {
"size": 16384,
"usage": 0
},
"stats": {
"size": 16384,
"usage": 0
},
"syscheck": {
"size": 16384,
"usage": 0
},
"syscollector": {
"size": 16384,
"usage": 0
},
"upgrade": {
"size": 16384,
"usage": 0
}
}
}
},
{
"uptime": "2024-05-10T15:39:45+00:00",
"timestamp": "2024-05-10T15:46:40+00:00",
"name": "wazuh-db",
"metrics": {
"queries": {
"received": 1811,
"received_breakdown": {
"agent": 220,
"agent_breakdown": {
"db": {
"begin": 0,
"close": 0,
"commit": 0,
"remove": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 2
},
"sca": {
"sca": 6
},
"sync": {
"dbsync": 0
},
"syscheck": {
"fim_file": 2,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 2
},
"syscollector": {
"syscollector_hotfixes": 1,
"syscollector_hwinfo": 2,
"syscollector_network_address": 1,
"syscollector_network_iface": 2,
"syscollector_network_protocol": 1,
"syscollector_osinfo": 2,
"syscollector_packages": 1,
"syscollector_ports": 24,
"syscollector_processes": 174,
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
}
}
}
},
"global": 188,
"global_breakdown": {
"db": {
"backup": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"agent": {
"delete-agent": 0,
"disconnect-agents": 0,
"find-agent": 0,
"get-agent-info": 0,
"get-agents-by-connection-status": 0,
"get-all-agents": 10,
"get-distinct-groups": 42,
"get-groups-integrity": 0,
"insert-agent": 0,
"reset-agents-connection": 1,
"select-agent-group": 0,
"select-agent-name": 1,
"set-agent-groups": 0,
"sync-agent-groups-get": 38,
"sync-agent-info-get": 0,
"sync-agent-info-set": 92,
"update-agent-data": 1,
"update-agent-name": 1,
"update-connection-status": 0,
"update-status-code": 0,
"update-keepalive": 0
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 0,
"find-group": 1,
"insert-agent-group": 0,
"select-groups": 1
},
"labels": {
"get-labels": 0
}
}
},
"mitre": 1400,
"mitre_breakdown": {
"db": {
"sql": 1400
}
},
"task": 3,
"task_breakdown": {
"db": {
"sql": 0
},
"tables": {
"tasks": {
"delete_old": 1,
"set_timeout": 1,
"upgrade": 0,
"upgrade_cancel_tasks": 1,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 0,
"wazuhdb_breakdown": {
"db": {
"remove": 0
}
}
}
},
"time": {
"execution": 2483,
"execution_breakdown": {
"agent": 42,
"agent_breakdown": {
"db": {
"open": 6,
"begin": 0,
"close": 0,
"commit": 0,
"remove": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 5
},
"sca": {
"sca": 2
},
"sync": {
"dbsync": 0
},
"syscheck": {
"fim_file": 1,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 8
},
"syscollector": {
"syscollector_hotfixes": 0,
"syscollector_hwinfo": 0,
"syscollector_network_address": 0,
"syscollector_network_iface": 0,
"syscollector_network_protocol": 0,
"syscollector_osinfo": 2,
"syscollector_packages": 0,
"syscollector_ports": 1,
"syscollector_processes": 13,
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
}
}
}
},
"global": 2341,
"global_breakdown": {
"db": {
"open": 66,
"backup": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"agent": {
"delete-agent": 0,
"disconnect-agents": 0,
"find-agent": 0,
"get-agent-info": 0,
"get-agents-by-connection-status": 0,
"get-all-agents": 649,
"get-distinct-groups": 760,
"get-groups-integrity": 0,
"insert-agent": 0,
"reset-agents-connection": 19,
"select-agent-group": 0,
"select-agent-name": 0,
"set-agent-groups": 0,
"sync-agent-groups-get": 539,
"sync-agent-info-get": 0,
"sync-agent-info-set": 305,
"update-agent-data": 0,
"update-agent-name": 0,
"update-connection-status": 0,
"update-status-code": 0,
"update-keepalive": 0
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 0,
"find-group": 0,
"insert-agent-group": 0,
"select-groups": 0
},
"labels": {
"get-labels": 0
}
}
},
"mitre": 100,
"mitre_breakdown": {
"db": {
"sql": 100
}
},
"task": 0,
"task_breakdown": {
"db": {
"sql": 0
},
"tables": {
"tasks": {
"delete_old": 0,
"set_timeout": 0,
"upgrade": 0,
"upgrade_cancel_tasks": 0,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 0,
"wazuhdb_breakdown": {
"db": {
"remove": 0
}
}
}
}
}
}
],
"total_affected_items": 3,
"total_failed_items": 0,
"failed_items": []
},
"message": "Statistical information for each daemon was successfully read",
"error": 0
}Worker 21curl -k -X GET "https://localhost:55000/cluster/CLUSTER-Workload_benchmarks_metrics_B517_manager_21/daemons/stats?pretty=true" -H "Authorization: Bearer $TOKEN"
{
"data": {
"affected_items": [
{
"uptime": "2024-05-10T15:14:01+00:00",
"timestamp": "2024-05-10T15:46:41+00:00",
"name": "wazuh-remoted",
"metrics": {
"bytes": {
"received": 70723932,
"sent": 2122183874
},
"keys_reload_count": 40,
"messages": {
"received_breakdown": {
"control": 59905,
"control_breakdown": {
"keepalive": 58051,
"request": 0,
"shutdown": 0,
"startup": 1858
},
"dequeued_after": 0,
"discarded": 70464,
"event": 0,
"ping": 0,
"unknown": 0
},
"sent_breakdown": {
"ack": 59909,
"ar": 5335,
"discarded": 1,
"request": 0,
"sca": 0,
"shared": 5367642
}
},
"queues": {
"received": {
"size": 131072,
"usage": 131071
}
},
"tcp_sessions": 0
}
},
{
"uptime": "2024-05-10T15:14:02+00:00",
"timestamp": "2024-05-10T15:46:41+00:00",
"name": "wazuh-analysisd",
"metrics": {
"bytes": {
"received": 686719
},
"events": {
"processed": 1891,
"received": 2838,
"received_breakdown": {
"decoded_breakdown": {
"agent": 0,
"agentless": 0,
"dbsync": 945,
"integrations_breakdown": {
"virustotal": 0
},
"modules_breakdown": {
"aws": 0,
"azure": 0,
"ciscat": 0,
"command": 0,
"docker": 0,
"gcp": 0,
"github": 0,
"logcollector_breakdown": {
"eventchannel": 0,
"eventlog": 0,
"macos": 0,
"others": 1886
},
"office365": 0,
"ms-graph": 0,
"oscap": 0,
"osquery": 0,
"rootcheck": 2,
"sca": 2,
"syscheck": 2,
"syscollector": 0,
"upgrade": 0,
"vulnerability": 0
},
"monitor": 1,
"remote": 0,
"syslog": 0
},
"dropped_breakdown": {
"agent": 0,
"agentless": 0,
"dbsync": 0,
"integrations_breakdown": {
"virustotal": 0
},
"modules_breakdown": {
"aws": 0,
"azure": 0,
"ciscat": 0,
"command": 0,
"docker": 0,
"gcp": 0,
"github": 0,
"logcollector_breakdown": {
"eventchannel": 0,
"eventlog": 0,
"macos": 0,
"others": 0
},
"office365": 0,
"ms-graph": 0,
"oscap": 0,
"osquery": 0,
"rootcheck": 0,
"sca": 0,
"syscheck": 0,
"syscollector": 0,
"upgrade": 0,
"vulnerability": 0
},
"monitor": 0,
"remote": 0,
"syslog": 0
}
},
"written_breakdown": {
"alerts": 112,
"archives": 0,
"firewall": 0,
"fts": 0,
"stats": 0
}
},
"queues": {
"alerts": {
"size": 16384,
"usage": 0
},
"archives": {
"size": 16384,
"usage": 0
},
"dbsync": {
"size": 16384,
"usage": 0
},
"eventchannel": {
"size": 16384,
"usage": 0
},
"firewall": {
"size": 16384,
"usage": 0
},
"fts": {
"size": 16384,
"usage": 0
},
"hostinfo": {
"size": 16384,
"usage": 0
},
"others": {
"size": 16384,
"usage": 0
},
"processed": {
"size": 16384,
"usage": 0
},
"rootcheck": {
"size": 16384,
"usage": 0
},
"sca": {
"size": 16384,
"usage": 0
},
"stats": {
"size": 16384,
"usage": 0
},
"syscheck": {
"size": 16384,
"usage": 0
},
"syscollector": {
"size": 16384,
"usage": 0
},
"upgrade": {
"size": 16384,
"usage": 0
}
}
}
},
{
"uptime": "2024-05-10T15:14:00+00:00",
"timestamp": "2024-05-10T15:46:41+00:00",
"name": "wazuh-db",
"metrics": {
"queries": {
"received": 153330,
"received_breakdown": {
"agent": 955,
"agent_breakdown": {
"db": {
"begin": 0,
"close": 0,
"commit": 0,
"remove": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 2
},
"sca": {
"sca": 6
},
"sync": {
"dbsync": 0
},
"syscheck": {
"fim_file": 43,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 2
},
"syscollector": {
"syscollector_hotfixes": 1,
"syscollector_hwinfo": 2,
"syscollector_network_address": 1,
"syscollector_network_iface": 2,
"syscollector_network_protocol": 1,
"syscollector_osinfo": 2,
"syscollector_packages": 626,
"syscollector_ports": 20,
"syscollector_processes": 247,
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
}
}
}
},
"global": 150080,
"global_breakdown": {
"db": {
"backup": 0,
"sql": 10,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"agent": {
"delete-agent": 895,
"disconnect-agents": 3,
"find-agent": 0,
"get-agent-info": 0,
"get-agents-by-connection-status": 324,
"get-all-agents": 256,
"get-distinct-groups": 196,
"get-groups-integrity": 0,
"insert-agent": 50003,
"reset-agents-connection": 1,
"select-agent-group": 36495,
"select-agent-name": 896,
"set-agent-groups": 180,
"sync-agent-groups-get": 206,
"sync-agent-info-get": 702,
"sync-agent-info-set": 0,
"update-agent-data": 36493,
"update-agent-name": 1,
"update-connection-status": 0,
"update-status-code": 0,
"update-keepalive": 23413
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 2,
"find-group": 2,
"insert-agent-group": 1,
"select-groups": 1
},
"labels": {
"get-labels": 0
}
}
},
"mitre": 1400,
"mitre_breakdown": {
"db": {
"sql": 1400
}
},
"task": 0,
"task_breakdown": {
"db": {
"sql": 0
},
"tables": {
"tasks": {
"delete_old": 0,
"set_timeout": 0,
"upgrade": 0,
"upgrade_cancel_tasks": 0,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 895,
"wazuhdb_breakdown": {
"db": {
"remove": 895
}
}
}
},
"time": {
"execution": 44402,
"execution_breakdown": {
"agent": 1609,
"agent_breakdown": {
"db": {
"open": 917,
"begin": 0,
"close": 0,
"commit": 0,
"remove": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 6
},
"sca": {
"sca": 2
},
"sync": {
"dbsync": 0
},
"syscheck": {
"fim_file": 48,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 4
},
"syscollector": {
"syscollector_hotfixes": 0,
"syscollector_hwinfo": 0,
"syscollector_network_address": 0,
"syscollector_network_iface": 2,
"syscollector_network_protocol": 1,
"syscollector_osinfo": 3,
"syscollector_packages": 599,
"syscollector_ports": 1,
"syscollector_processes": 22,
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
}
}
}
},
"global": 42671,
"global_breakdown": {
"db": {
"open": 5421,
"backup": 0,
"sql": 36,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"agent": {
"delete-agent": 99,
"disconnect-agents": 24,
"find-agent": 0,
"get-agent-info": 0,
"get-agents-by-connection-status": 4353,
"get-all-agents": 13861,
"get-distinct-groups": 1458,
"get-groups-integrity": 0,
"insert-agent": 1173,
"reset-agents-connection": 0,
"select-agent-group": 798,
"select-agent-name": 12,
"set-agent-groups": 7277,
"sync-agent-groups-get": 1395,
"sync-agent-info-get": 2624,
"sync-agent-info-set": 0,
"update-agent-data": 2971,
"update-agent-name": 0,
"update-connection-status": 0,
"update-status-code": 0,
"update-keepalive": 1015
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 146,
"find-group": 0,
"insert-agent-group": 0,
"select-groups": 0
},
"labels": {
"get-labels": 0
}
}
},
"mitre": 114,
"mitre_breakdown": {
"db": {
"sql": 114
}
},
"task": 0,
"task_breakdown": {
"db": {
"sql": 0
},
"tables": {
"tasks": {
"delete_old": 0,
"set_timeout": 0,
"upgrade": 0,
"upgrade_cancel_tasks": 0,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 8,
"wazuhdb_breakdown": {
"db": {
"remove": 8
}
}
}
}
}
}
],
"total_affected_items": 3,
"total_failed_items": 0,
"failed_items": []
},
"message": "Statistical information for each daemon was successfully read",
"error": 0
}Build 518 (v4.8.0-rc1)Artifacts: artifacts_480_rc1_B518.zip
Tasks stats>>> from wazuh_testing.tools.performance.csv_parser import ClusterCSVTasksParser
>>> ClusterCSVTasksParser('/home/selu/Descargas/cluster_performance/518/artifacts_480_rc1_B518').get_stats()
{
"setup_phase": {
"integrity_check": {
"time_spent(s)": {
"workers": {
"mean": ("worker_23", 0.3483846153846154),
"max": ("worker_14", 2.27),
},
"master": {
"mean": ("master", 0.023405487804878052),
"max": ("master", 0.34),
},
}
},
"integrity_sync": {
"time_spent(s)": {
"workers": {
"mean": ("worker_1", 0.04331111111111111),
"max": ("worker_17", 0.151),
},
"master": {
"mean": ("master", 0.40935483870967737),
"max": ("master", 2.593),
},
}
},
"agent-info_sync": {
"time_spent(s)": {
"workers": {
"mean": ("worker_22", 0.37691525423728817),
"max": ("worker_22", 3.229),
},
"master": {
"mean": ("master", 0.19893548387096774),
"max": ("master", 2.626),
},
}
},
},
"stable_phase": {
"integrity_check": {
"time_spent(s)": {
"workers": {
"mean": ("worker_1", 0.21219101123595507),
"max": ("worker_6", 1.091),
},
"master": {
"mean": ("master", 0.09881998199819982),
"max": ("master", 0.56),
},
}
},
"agent-info_sync": {
"time_spent(s)": {
"workers": {
"mean": ("worker_9", 2.959135802469136),
"max": ("worker_17", 25.024),
},
"master": {
"mean": ("master", 2.276723549488054),
"max": ("master", 24.654),
},
}
},
},
}Resources stats>>> from wazuh_testing.tools.performance.csv_parser import ClusterCSVResourcesParser
>>> ClusterCSVResourcesParser('/home/selu/Descargas/cluster_performance/518/artifacts_480_rc1_B518').get_stats()
{
"setup_phase": {
"wazuh-clusterd": {
"USS(KB)": {
"workers": {
"mean": ("worker_1", 47902.813559322036),
"max": ("worker_1", 85892.0),
"reg_cof": ("worker_8", 439.21965414264884),
},
"master": {
"mean": ("master", 58250.57627118644),
"max": ("master", 92504.0),
"reg_cof": ("master", 331.84353167603393),
},
},
"CPU(%)": {
"workers": {
"mean": ("worker_1", 1.9576271186440684),
"max": ("worker_6", 4.6),
"reg_cof": ("worker_6", 0.008374144964374271),
},
"master": {
"mean": ("master", 54.65),
"max": ("master", 153.7),
"reg_cof": ("master", 0.5279038342846917),
},
},
"FD": {
"workers": {
"mean": ("worker_11", 73.61344537815125),
"max": ("worker_12", 76),
"reg_cof": ("worker_8", 0.05867074102368176),
},
"master": {
"mean": ("master", 136.8728813559322),
"max": ("master", 140),
"reg_cof": ("master", 0.02001687245954424),
},
},
}
},
"stable_phase": {
"wazuh-clusterd": {
"USS(KB)": {
"workers": {
"mean": ("worker_8", 66593.85792349727),
"max": ("worker_19", 82216.0),
"reg_cof": ("worker_21", 104.67777994449867),
},
"master": {
"mean": ("master", 91653.54347826086),
"max": ("master", 151604.0),
"reg_cof": ("master", 317.8893683419697),
},
},
"CPU(%)": {
"workers": {
"mean": ("worker_1", 0.9573770491803278),
"max": ("worker_21", 4.4),
"reg_cof": ("worker_21", -0.0033231830795769903),
},
"master": {
"mean": ("master", 14.179891304347828),
"max": ("master", 145.2),
"reg_cof": ("master", -0.0822202423378475),
},
},
"FD": {
"workers": {
"mean": ("worker_10", 73.78142076502732),
"max": ("worker_19", 75),
"reg_cof": ("worker_21", 0.09893722343058586),
},
"master": {
"mean": ("master", 136.32065217391303),
"max": ("master", 140),
"reg_cof": ("master", 0.016669877289977156),
},
},
}
},
}v4.7.3For comparison, these were the results obtained in the workload benchmark of v4.7.3.
Tasks stats>>> from wazuh_testing.tools.performance.csv_parser import ClusterCSVTasksParser
>>> ClusterCSVTasksParser('/home/selu/Descargas/cluster_performance/v473/artifacts_473').get_stats()
{
"setup_phase": {
"integrity_check": {
"time_spent(s)": {
"workers": {
"mean": ("worker_19", 0.08235999999999997),
"max": ("worker_19", 0.619),
},
"master": {
"mean": ("master", 0.006041507024265645),
"max": ("master", 0.075),
},
}
},
"integrity_sync": {
"time_spent(s)": {
"workers": {
"mean": ("worker_22", 0.03858695652173912),
"max": ("worker_22", 0.085),
},
"master": {
"mean": ("master", 0.2113018056749785),
"max": ("master", 0.707),
},
}
},
"agent-info_sync": {
"time_spent(s)": {
"workers": {
"mean": ("worker_1", 0.20336036036036037),
"max": ("worker_11", 1.041),
},
"master": {
"mean": ("master", 0.08882754759238522),
"max": ("master", 0.914),
},
}
},
},
"stable_phase": {
"integrity_check": {
"time_spent(s)": {
"workers": {
"mean": ("worker_22", 0.010945945945945947),
"max": ("worker_11", 0.035),
},
"master": {
"mean": ("master", 0.004536216216216217),
"max": ("master", 0.023),
},
}
},
"agent-info_sync": {
"time_spent(s)": {
"workers": {
"mean": ("worker_25", 0.030285714285714267),
"max": ("worker_25", 1.018),
},
"master": {
"mean": ("master", 0.31563157894736843),
"max": ("master", 0.48),
},
}
},
},
}Resources stats>>> from wazuh_testing.tools.performance.csv_parser import ClusterCSVResourcesParser
>>> ClusterCSVResourcesParser('/home/selu/Descargas/cluster_performance/v473/artifacts_473').get_stats()
{
"setup_phase": {
"wazuh-clusterd": {
"USS(KB)": {
"workers": {
"mean": ("worker_8", 66134.16393442624),
"max": ("worker_3", 84572.0),
"reg_cof": ("worker_4", 118.61238524998139),
},
"master": {
"mean": ("master", 50613.98275862069),
"max": ("master", 76240.0),
"reg_cof": ("master", 107.4721952607109),
},
},
"CPU(%)": {
"workers": {
"mean": ("worker_8", 1.6155737704918034),
"max": ("worker_9", 10.2),
"reg_cof": ("worker_6", -0.008092470746647176),
},
"master": {
"mean": ("master", 37.39525862068966),
"max": ("master", 140.8),
"reg_cof": ("master", -0.18205761605569198),
},
},
"FD": {
"workers": {
"mean": ("worker_7", 74.5),
"max": ("worker_16", 76),
"reg_cof": ("worker_25", 0.01310659409705495),
},
"master": {
"mean": ("master", 133.10344827586206),
"max": ("master", 161.0),
"reg_cof": ("master", -0.0031030830942030637),
},
},
}
},
"stable_phase": {
"wazuh-clusterd": {
"USS(KB)": {
"workers": {
"mean": ("worker_24", 79203.88405797101),
"max": ("worker_10", 83228.0),
"reg_cof": ("worker_3", 607.4142973412916),
},
"master": {
"mean": ("master", 98324.22222222222),
"max": ("master", 102912.0),
"reg_cof": ("master", 188.28207601775017),
},
},
"CPU(%)": {
"workers": {
"mean": ("worker_10", 0.46760563380281683),
"max": ("worker_10", 15.0),
"reg_cof": ("worker_3", 0.0009658166033640807),
},
"master": {
"mean": ("master", 2.163888888888889),
"max": ("master", 34.4),
"reg_cof": ("master", -0.07464467168306647),
},
},
"FD": {
"workers": {
"mean": ("worker_16", 76.02857142857142),
"max": ("worker_16", 77),
"reg_cof": ("worker_3", 0.584902333152469),
},
"master": {
"mean": ("master", 132.19444444444446),
"max": ("master", 136.0),
"reg_cof": ("master", 0.2557077625570762),
},
},
}
},
}4.7.3 vs 4.8.0If we compare the results of 4.7.3 and build 517 (4.8.0), even though the stats were not high enough to exceed the threshold and make the test fail in said build, we can see how the times are clearly greater:
I will be running a v4.7.4 build to get its wazuh-db stats and compare it with 4.8.0. v4.7.3{
"setup_phase": {
"agent-info_sync": {
"time_spent(s)": {
"workers": {
"mean": ("worker_1", 0.20336036036036037),
"max": ("worker_11", 1.041),
},
"master": {
"mean": ("master", 0.08882754759238522),
"max": ("master", 0.914),
},
}
},
}
}4.8.0-rc1{
"setup_phase": {
"agent-info_sync": {
"time_spent(s)": {
"workers": {
"mean": ("worker_18", 0.9509827586206897),
"max": ("worker_9", 10.639),
},
"master": {
"mean": ("master", 0.687005693950178),
"max": ("master", 10.257),
},
}
}
}
} |
Build 521 (v4.7.4)Artifacts: artifacts_v474.zip
As can be seen above, the test failed due to the number of file descriptors used by the clusterd process exceeding the threshold, which is not too worrying since these thresholds still need to be refined for resource usage. All was fine regarding cluster tasks duration, especially after breaking down the stats: Tasks stats>>> from wazuh_testing.tools.performance.csv_parser import ClusterCSVTasksParser
>>> ClusterCSVTasksParser('/home/selu/Descargas/cluster_performance/521/artifacts_v474').get_stats()
{
"setup_phase": {
"integrity_check": {
"time_spent(s)": {
"workers": {
"mean": ("worker_23", 0.1903700787401575),
"max": ("worker_21", 2.563),
},
"master": {
"mean": ("master", 0.008502512562814072),
"max": ("master", 0.43),
},
}
},
"integrity_sync": {
"time_spent(s)": {
"workers": {
"mean": ("worker_12", 0.03845652173913043),
"max": ("worker_16", 0.098),
},
"master": {
"mean": ("master", 0.3908857630008526),
"max": ("master", 2.629),
},
}
},
"agent-info_sync": {
"time_spent(s)": {
"workers": {
"mean": ("worker_6", 0.2628070175438596),
"max": ("worker_16", 2.468),
},
"master": {
"mean": ("master", 0.15243092707951633),
"max": ("master", 2.33),
},
}
},
},
"stable_phase": {
"integrity_check": {
"time_spent(s)": {
"workers": {
"mean": ("worker_24", 0.01088888888888889),
"max": ("worker_24", 0.035),
},
"master": {
"mean": ("master", 0.004245925925925926),
"max": ("master", 0.022),
},
}
},
"agent-info_sync": {
"time_spent(s)": {
"workers": {"mean": ("worker_5", 0.01084), "max": ("worker_5", 0.229)},
"master": {"mean": ("master", 0.0425), "max": ("master", 0.066)},
}
},
},
}Resources stats>>> from wazuh_testing.tools.performance.csv_parser import ClusterCSVResourcesParser
>>> ClusterCSVResourcesParser('/home/selu/Descargas/cluster_performance/521/artifacts_v474').get_stats()
{
"setup_phase": {
"wazuh-clusterd": {
"USS(KB)": {
"workers": {
"mean": ("worker_21", 91373.91235059762),
"max": ("worker_21", 148064.0),
"reg_cof": ("worker_21", 310.1989482071715),
},
"master": {
"mean": ("master", 57898.19913419913),
"max": ("master", 129064.0),
"reg_cof": ("master", 176.50359625382117),
},
},
"CPU(%)": {
"workers": {
"mean": ("worker_22", 1.6698412698412701),
"max": ("worker_7", 20.8),
"reg_cof": ("worker_25", -0.007463239457003537),
},
"master": {
"mean": ("master", 41.56796536796537),
"max": ("master", 194.4),
"reg_cof": ("master", -0.22277818882766384),
},
},
"FD": {
"workers": {
"mean": ("worker_21", 113.9601593625498),
"max": ("worker_21", 117.0),
"reg_cof": ("worker_22", -0.007738292079189382),
},
"master": {
"mean": ("master", 134.995670995671),
"max": ("master", 142.0),
"reg_cof": ("master", -0.04760996125313914),
},
},
}
},
"stable_phase": {
"wazuh-clusterd": {
"USS(KB)": {
"workers": {
"mean": ("worker_17", 63396.705882352944),
"max": ("worker_17", 63428.0),
"reg_cof": ("worker_22", 87.98292495517883),
}
},
"CPU(%)": {
"workers": {
"mean": ("worker_21", 0.33076923076923076),
"max": ("worker_17", 10.0),
"reg_cof": ("worker_4", 0.0008451380552220902),
}
},
"FD": {
"workers": {
"mean": ("worker_21", 104.5),
"max": ("worker_21", 116.0),
"reg_cof": ("worker_21", 0.9427132246222142),
}
},
}
},
}Services statsBefore restartingThe master node was restarted at MASTERwazuh-db{
"uptime": "2024-05-13T11:41:52+00:00",
"timestamp": "2024-05-13T12:06:44+00:00",
"name": "wazuh-db",
"metrics": {
"queries": {
"received": 347321,
"received_breakdown": {
"agent": 955,
"agent_breakdown": {
"db": {
"begin": 0,
"close": 0,
"commit": 0,
"remove": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 2
},
"sca": {
"sca": 6
},
"sync": {
"dbsync": 0
},
"syscheck": {
"fim_file": 64,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 2
},
"syscollector": {
"syscollector_hotfixes": 1,
"syscollector_hwinfo": 2,
"syscollector_network_address": 1,
"syscollector_network_iface": 2,
"syscollector_network_protocol": 1,
"syscollector_osinfo": 2,
"syscollector_packages": 597,
"syscollector_ports": 15,
"syscollector_processes": 260,
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
}
},
"vulnerability": {
"vuln_cves": 0
}
}
},
"global": 344139,
"global_breakdown": {
"db": {
"backup": 0,
"sql": 455,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"agent": {
"delete-agent": 571,
"disconnect-agents": 2,
"find-agent": 0,
"get-agent-info": 150003,
"get-agents-by-connection-status": 0,
"get-all-agents": 1,
"get-distinct-groups": 150,
"get-groups-integrity": 0,
"insert-agent": 50003,
"reset-agents-connection": 1,
"select-agent-group": 0,
"select-agent-name": 0,
"set-agent-groups": 92325,
"sync-agent-groups-get": 328,
"sync-agent-info-get": 0,
"sync-agent-info-set": 50291,
"update-agent-data": 1,
"update-agent-name": 1,
"update-connection-status": 0,
"update-status-code": 0,
"update-keepalive": 0
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 3,
"find-group": 2,
"insert-agent-group": 1,
"select-groups": 1
},
"labels": {
"get-labels": 0
}
}
},
"mitre": 1625,
"mitre_breakdown": {
"db": {
"sql": 1625
}
},
"task": 31,
"task_breakdown": {
"db": {
"sql": 2
},
"tables": {
"tasks": {
"delete_old": 1,
"set_timeout": 2,
"upgrade": 0,
"upgrade_cancel_tasks": 26,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 571,
"wazuhdb_breakdown": {
"db": {
"remove": 571
}
}
}
},
"time": {
"execution": 288324,
"execution_breakdown": {
"agent": 165,
"agent_breakdown": {
"db": {
"open": 21,
"begin": 0,
"close": 0,
"commit": 0,
"remove": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 4
},
"sca": {
"sca": 0
},
"sync": {
"dbsync": 0
},
"syscheck": {
"fim_file": 12,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 4
},
"syscollector": {
"syscollector_hotfixes": 0,
"syscollector_hwinfo": 0,
"syscollector_network_address": 0,
"syscollector_network_iface": 0,
"syscollector_network_protocol": 0,
"syscollector_osinfo": 0,
"syscollector_packages": 104,
"syscollector_ports": 0,
"syscollector_processes": 16,
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
}
},
"vulnerability": {
"vuln_cves": 0
}
}
},
"global": 287285,
"global_breakdown": {
"db": {
"open": 87887,
"backup": 0,
"sql": 4284,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"agent": {
"delete-agent": 132,
"disconnect-agents": 29,
"find-agent": 0,
"get-agent-info": 3176,
"get-agents-by-connection-status": 0,
"get-all-agents": 0,
"get-distinct-groups": 2024,
"get-groups-integrity": 0,
"insert-agent": 2940,
"reset-agents-connection": 0,
"select-agent-group": 0,
"select-agent-name": 0,
"set-agent-groups": 6464,
"sync-agent-groups-get": 6497,
"sync-agent-info-get": 0,
"sync-agent-info-set": 173841,
"update-agent-data": 0,
"update-agent-name": 0,
"update-connection-status": 0,
"update-status-code": 0,
"update-keepalive": 0
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 6,
"find-group": 0,
"insert-agent-group": 0,
"select-groups": 0
},
"labels": {
"get-labels": 0
}
}
},
"mitre": 864,
"mitre_breakdown": {
"db": {
"sql": 864
}
},
"task": 1,
"task_breakdown": {
"db": {
"sql": 0
},
"tables": {
"tasks": {
"delete_old": 0,
"set_timeout": 0,
"upgrade": 0,
"upgrade_cancel_tasks": 1,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 9,
"wazuhdb_breakdown": {
"db": {
"remove": 9
}
}
}
}
}
}wazuh-remoted{
"uptime": "2024-05-13T11:41:53+00:00",
"timestamp": "2024-05-13T12:06:44+00:00",
"name": "wazuh-remoted",
"metrics": {
"bytes": {
"received": 0,
"sent": 0
},
"keys_reload_count": 42,
"messages": {
"received_breakdown": {
"control": 0,
"control_breakdown": {
"keepalive": 0,
"request": 0,
"shutdown": 0,
"startup": 0
},
"dequeued_after": 0,
"discarded": 0,
"event": 0,
"ping": 0,
"unknown": 0
},
"sent_breakdown": {
"ack": 0,
"ar": 0,
"discarded": 0,
"request": 0,
"sca": 0,
"shared": 0
}
},
"queues": {
"received": {
"size": 131072,
"usage": 0
}
},
"tcp_sessions": 0
}
}WORKER 21wazuh-db{
"uptime": "2024-05-13T11:42:02+00:00",
"timestamp": "2024-05-13T12:06:44+00:00",
"name": "wazuh-db",
"metrics": {
"queries": {
"received": 386447,
"received_breakdown": {
"agent": 929,
"agent_breakdown": {
"db": {
"begin": 0,
"close": 0,
"commit": 0,
"remove": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 2
},
"sca": {
"sca": 6
},
"sync": {
"dbsync": 0
},
"syscheck": {
"fim_file": 41,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 2
},
"syscollector": {
"syscollector_hotfixes": 1,
"syscollector_hwinfo": 2,
"syscollector_network_address": 1,
"syscollector_network_iface": 2,
"syscollector_network_protocol": 1,
"syscollector_osinfo": 2,
"syscollector_packages": 597,
"syscollector_ports": 17,
"syscollector_processes": 255,
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
}
},
"vulnerability": {
"vuln_cves": 0
}
}
},
"global": 383523,
"global_breakdown": {
"db": {
"backup": 0,
"sql": 9,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"agent": {
"delete-agent": 667,
"disconnect-agents": 2,
"find-agent": 0,
"get-agent-info": 0,
"get-agents-by-connection-status": 217,
"get-all-agents": 246,
"get-distinct-groups": 149,
"get-groups-integrity": 0,
"insert-agent": 50003,
"reset-agents-connection": 1,
"select-agent-group": 162616,
"select-agent-name": 667,
"set-agent-groups": 146,
"sync-agent-groups-get": 172,
"sync-agent-info-get": 2077,
"sync-agent-info-set": 0,
"update-agent-data": 162616,
"update-agent-name": 1,
"update-connection-status": 0,
"update-status-code": 0,
"update-keepalive": 3929
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 1,
"find-group": 2,
"insert-agent-group": 1,
"select-groups": 1
},
"labels": {
"get-labels": 0
}
}
},
"mitre": 1328,
"mitre_breakdown": {
"db": {
"sql": 1328
}
},
"task": 0,
"task_breakdown": {
"db": {
"sql": 0
},
"tables": {
"tasks": {
"delete_old": 0,
"set_timeout": 0,
"upgrade": 0,
"upgrade_cancel_tasks": 0,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 667,
"wazuhdb_breakdown": {
"db": {
"remove": 667
}
}
}
},
"time": {
"execution": 58148,
"execution_breakdown": {
"agent": 167,
"agent_breakdown": {
"db": {
"open": 23,
"begin": 0,
"close": 0,
"commit": 0,
"remove": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 2
},
"sca": {
"sca": 0
},
"sync": {
"dbsync": 0
},
"syscheck": {
"fim_file": 9,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 4
},
"syscollector": {
"syscollector_hotfixes": 0,
"syscollector_hwinfo": 0,
"syscollector_network_address": 0,
"syscollector_network_iface": 0,
"syscollector_network_protocol": 0,
"syscollector_osinfo": 0,
"syscollector_packages": 106,
"syscollector_ports": 1,
"syscollector_processes": 16,
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
}
},
"vulnerability": {
"vuln_cves": 0
}
}
},
"global": 57857,
"global_breakdown": {
"db": {
"open": 16127,
"backup": 0,
"sql": 41,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"agent": {
"delete-agent": 74,
"disconnect-agents": 18,
"find-agent": 0,
"get-agent-info": 0,
"get-agents-by-connection-status": 3426,
"get-all-agents": 13818,
"get-distinct-groups": 1253,
"get-groups-integrity": 0,
"insert-agent": 1066,
"reset-agents-connection": 0,
"select-agent-group": 2940,
"select-agent-name": 8,
"set-agent-groups": 1482,
"sync-agent-groups-get": 947,
"sync-agent-info-get": 4985,
"sync-agent-info-set": 0,
"update-agent-data": 11496,
"update-agent-name": 0,
"update-connection-status": 0,
"update-status-code": 0,
"update-keepalive": 169
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 2,
"find-group": 0,
"insert-agent-group": 0,
"select-groups": 0
},
"labels": {
"get-labels": 0
}
}
},
"mitre": 119,
"mitre_breakdown": {
"db": {
"sql": 119
}
},
"task": 0,
"task_breakdown": {
"db": {
"sql": 0
},
"tables": {
"tasks": {
"delete_old": 0,
"set_timeout": 0,
"upgrade": 0,
"upgrade_cancel_tasks": 0,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 5,
"wazuhdb_breakdown": {
"db": {
"remove": 5
}
}
}
}
}
}wazuh-remoted{
"uptime": "2024-05-13T11:42:03+00:00",
"timestamp": "2024-05-13T12:06:44+00:00",
"name": "wazuh-remoted",
"metrics": {
"bytes": {
"received": 44928111,
"sent": 1273761881
},
"keys_reload_count": 35,
"messages": {
"received_breakdown": {
"control": 166544,
"control_breakdown": {
"keepalive": 164527,
"request": 0,
"shutdown": 0,
"startup": 2018
},
"dequeued_after": 0,
"discarded": 0,
"event": 0,
"ping": 0,
"unknown": 0
},
"sent_breakdown": {
"ack": 166545,
"ar": 6054,
"discarded": 26,
"request": 0,
"sca": 0,
"shared": 3162326
}
},
"queues": {
"received": {
"size": 131072,
"usage": 0
}
},
"tcp_sessions": 2018
}
}After restartingThese stats are right before the hosts were destroyed: MASTERwazuh-db{
"uptime": "2024-05-13T12:08:17+00:00",
"timestamp": "2024-05-13T12:16:06+00:00",
"name": "wazuh-db",
"metrics": {
"queries": {
"received": 4981,
"received_breakdown": {
"agent": 261,
"agent_breakdown": {
"db": {
"begin": 0,
"close": 0,
"commit": 0,
"remove": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 2
},
"sca": {
"sca": 6
},
"sync": {
"dbsync": 0
},
"syscheck": {
"fim_file": 2,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 2
},
"syscollector": {
"syscollector_hotfixes": 1,
"syscollector_hwinfo": 2,
"syscollector_network_address": 1,
"syscollector_network_iface": 2,
"syscollector_network_protocol": 1,
"syscollector_osinfo": 2,
"syscollector_packages": 1,
"syscollector_ports": 19,
"syscollector_processes": 220,
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
}
},
"vulnerability": {
"vuln_cves": 0
}
}
},
"global": 3364,
"global_breakdown": {
"db": {
"backup": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"agent": {
"delete-agent": 0,
"disconnect-agents": 0,
"find-agent": 0,
"get-agent-info": 0,
"get-agents-by-connection-status": 0,
"get-all-agents": 1,
"get-distinct-groups": 47,
"get-groups-integrity": 0,
"insert-agent": 0,
"reset-agents-connection": 1,
"select-agent-group": 0,
"select-agent-name": 0,
"set-agent-groups": 2891,
"sync-agent-groups-get": 44,
"sync-agent-info-get": 0,
"sync-agent-info-set": 376,
"update-agent-data": 1,
"update-agent-name": 1,
"update-connection-status": 0,
"update-status-code": 0,
"update-keepalive": 0
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 0,
"find-group": 1,
"insert-agent-group": 0,
"select-groups": 1
},
"labels": {
"get-labels": 0
}
}
},
"mitre": 1328,
"mitre_breakdown": {
"db": {
"sql": 1328
}
},
"task": 28,
"task_breakdown": {
"db": {
"sql": 0
},
"tables": {
"tasks": {
"delete_old": 1,
"set_timeout": 1,
"upgrade": 0,
"upgrade_cancel_tasks": 26,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 0,
"wazuhdb_breakdown": {
"db": {
"remove": 0
}
}
}
},
"time": {
"execution": 691,
"execution_breakdown": {
"agent": 39,
"agent_breakdown": {
"db": {
"open": 1,
"begin": 0,
"close": 0,
"commit": 0,
"remove": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 5
},
"sca": {
"sca": 0
},
"sync": {
"dbsync": 0
},
"syscheck": {
"fim_file": 1,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 13
},
"syscollector": {
"syscollector_hotfixes": 0,
"syscollector_hwinfo": 0,
"syscollector_network_address": 0,
"syscollector_network_iface": 0,
"syscollector_network_protocol": 0,
"syscollector_osinfo": 0,
"syscollector_packages": 0,
"syscollector_ports": 1,
"syscollector_processes": 14,
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
}
},
"vulnerability": {
"vuln_cves": 0
}
}
},
"global": 547,
"global_breakdown": {
"db": {
"open": 13,
"backup": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"agent": {
"delete-agent": 0,
"disconnect-agents": 0,
"find-agent": 0,
"get-agent-info": 0,
"get-agents-by-connection-status": 0,
"get-all-agents": 0,
"get-distinct-groups": 1,
"get-groups-integrity": 0,
"insert-agent": 0,
"reset-agents-connection": 0,
"select-agent-group": 0,
"select-agent-name": 0,
"set-agent-groups": 266,
"sync-agent-groups-get": 1,
"sync-agent-info-get": 0,
"sync-agent-info-set": 264,
"update-agent-data": 0,
"update-agent-name": 0,
"update-connection-status": 0,
"update-status-code": 0,
"update-keepalive": 0
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 0,
"find-group": 0,
"insert-agent-group": 0,
"select-groups": 0
},
"labels": {
"get-labels": 0
}
}
},
"mitre": 104,
"mitre_breakdown": {
"db": {
"sql": 104
}
},
"task": 1,
"task_breakdown": {
"db": {
"sql": 0
},
"tables": {
"tasks": {
"delete_old": 0,
"set_timeout": 0,
"upgrade": 0,
"upgrade_cancel_tasks": 0,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 0,
"wazuhdb_breakdown": {
"db": {
"remove": 0
}
}
}
}
}
}wazuh-remoted{
"uptime": "2024-05-13T12:08:19+00:00",
"timestamp": "2024-05-13T12:16:06+00:00",
"name": "wazuh-remoted",
"metrics": {
"bytes": {
"received": 0,
"sent": 0
},
"keys_reload_count": 0,
"messages": {
"received_breakdown": {
"control": 0,
"control_breakdown": {
"keepalive": 0,
"request": 0,
"shutdown": 0,
"startup": 0
},
"dequeued_after": 0,
"discarded": 0,
"event": 0,
"ping": 0,
"unknown": 0
},
"sent_breakdown": {
"ack": 0,
"ar": 0,
"discarded": 0,
"request": 0,
"sca": 0,
"shared": 0
}
},
"queues": {
"received": {
"size": 131072,
"usage": 0
}
},
"tcp_sessions": 0
}
}WORKER 21wazuh-db{
"uptime": "2024-05-13T12:08:16+00:00",
"timestamp": "2024-05-13T12:16:06+00:00",
"name": "wazuh-db",
"metrics": {
"queries": {
"received": 158806,
"received_breakdown": {
"agent": 293,
"agent_breakdown": {
"db": {
"begin": 0,
"close": 0,
"commit": 0,
"remove": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 2
},
"sca": {
"sca": 6
},
"sync": {
"dbsync": 0
},
"syscheck": {
"fim_file": 27,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 2
},
"syscollector": {
"syscollector_hotfixes": 1,
"syscollector_hwinfo": 2,
"syscollector_network_address": 1,
"syscollector_network_iface": 2,
"syscollector_network_protocol": 1,
"syscollector_osinfo": 2,
"syscollector_packages": 1,
"syscollector_ports": 21,
"syscollector_processes": 225,
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
}
},
"vulnerability": {
"vuln_cves": 0
}
}
},
"global": 107849,
"global_breakdown": {
"db": {
"backup": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"agent": {
"delete-agent": 49336,
"disconnect-agents": 0,
"find-agent": 0,
"get-agent-info": 0,
"get-agents-by-connection-status": 7,
"get-all-agents": 20,
"get-distinct-groups": 47,
"get-groups-integrity": 0,
"insert-agent": 0,
"reset-agents-connection": 1,
"select-agent-group": 4469,
"select-agent-name": 49336,
"set-agent-groups": 44,
"sync-agent-groups-get": 44,
"sync-agent-info-get": 72,
"sync-agent-info-set": 0,
"update-agent-data": 4470,
"update-agent-name": 1,
"update-connection-status": 0,
"update-status-code": 0,
"update-keepalive": 0
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 0,
"find-group": 1,
"insert-agent-group": 0,
"select-groups": 1
},
"labels": {
"get-labels": 0
}
}
},
"mitre": 1328,
"mitre_breakdown": {
"db": {
"sql": 1328
}
},
"task": 0,
"task_breakdown": {
"db": {
"sql": 0
},
"tables": {
"tasks": {
"delete_old": 0,
"set_timeout": 0,
"upgrade": 0,
"upgrade_cancel_tasks": 0,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 49336,
"wazuhdb_breakdown": {
"db": {
"remove": 49336
}
}
}
},
"time": {
"execution": 5040,
"execution_breakdown": {
"agent": 38,
"agent_breakdown": {
"db": {
"open": 2,
"begin": 0,
"close": 0,
"commit": 0,
"remove": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 3
},
"sca": {
"sca": 0
},
"sync": {
"dbsync": 0
},
"syscheck": {
"fim_file": 8,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 3
},
"syscollector": {
"syscollector_hotfixes": 0,
"syscollector_hwinfo": 0,
"syscollector_network_address": 0,
"syscollector_network_iface": 0,
"syscollector_network_protocol": 0,
"syscollector_osinfo": 0,
"syscollector_packages": 1,
"syscollector_ports": 1,
"syscollector_processes": 15,
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
}
},
"vulnerability": {
"vuln_cves": 0
}
}
},
"global": 4551,
"global_breakdown": {
"db": {
"open": 939,
"backup": 0,
"sql": 0,
"vacuum": 0,
"get_fragmentation": 0,
"sleep": 0
},
"tables": {
"agent": {
"delete-agent": 1279,
"disconnect-agents": 0,
"find-agent": 0,
"get-agent-info": 0,
"get-agents-by-connection-status": 102,
"get-all-agents": 1200,
"get-distinct-groups": 97,
"get-groups-integrity": 0,
"insert-agent": 0,
"reset-agents-connection": 103,
"select-agent-group": 90,
"select-agent-name": 275,
"set-agent-groups": 0,
"sync-agent-groups-get": 1,
"sync-agent-info-get": 65,
"sync-agent-info-set": 0,
"update-agent-data": 394,
"update-agent-name": 0,
"update-connection-status": 0,
"update-status-code": 0,
"update-keepalive": 0
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 0,
"find-group": 0,
"insert-agent-group": 0,
"select-groups": 0
},
"labels": {
"get-labels": 0
}
}
},
"mitre": 111,
"mitre_breakdown": {
"db": {
"sql": 111
}
},
"task": 0,
"task_breakdown": {
"db": {
"sql": 0
},
"tables": {
"tasks": {
"delete_old": 0,
"set_timeout": 0,
"upgrade": 0,
"upgrade_cancel_tasks": 0,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 340,
"wazuhdb_breakdown": {
"db": {
"remove": 340
}
}
}
}
}
}wazuh-remoted{
"uptime": "2024-05-13T12:08:18+00:00",
"timestamp": "2024-05-13T12:16:06+00:00",
"name": "wazuh-remoted",
"metrics": {
"bytes": {
"received": 11209220,
"sent": 397741
},
"keys_reload_count": 1,
"messages": {
"received_breakdown": {
"control": 4469,
"control_breakdown": {
"keepalive": 4469,
"request": 0,
"shutdown": 0,
"startup": 0
},
"dequeued_after": 0,
"discarded": 0,
"event": 0,
"ping": 0,
"unknown": 36774
},
"sent_breakdown": {
"ack": 4469,
"ar": 0,
"discarded": 0,
"request": 0,
"sca": 0,
"shared": 0
}
},
"queues": {
"received": {
"size": 131072,
"usage": 0
}
},
"tcp_sessions": 0
}
}Full stats can be downloaded here.
|
| Query type | Number | Time (ms) | Time per request (ms) |
|---|---|---|---|
| received | 297596 | 1586000 (26.43 min) | 5.33 |
| global | 294874 | 1584778 (26.41 min) | 5.37 |
| open | - | 869160 (14.49 min) | - |
| get-agent-info | 150003 | 3858 (0.06 min) | 0.025 |
| set-agent-groups | 79152 | 624287 (10.40 min) | 7.88 |
| sync-agent-info-set | 14788 | 72447 (1.20 min) | 4.9 |
wazuh-db stats (4.7.4)
This was the uptime (24.87 min -> 1492 seconds -> 1492000ms):
"uptime": "2024-05-13T11:41:52+00:00",
"timestamp": "2024-05-13T12:06:44+00:00",
| Query type | Number | Time (ms) | Time per request (ms) |
|---|---|---|---|
| received | 347321 | 288324 (4.8 min) | 0.83 |
| global | 344139 | 287285 (4.78 min) | 0.83 |
| open | - | 87887 (1.4 min) | - |
| get-agent-info | 150003 | 3176 (0.05 min) | 0.02 |
| set-agent-groups | 92325 | 6464 (0.1 min) | 0.07 |
| sync-agent-info-set | 50291 | 173841 (2.89 min) | 3.45 |
4.7.3 vs 4.7.4 vs 4.8.0
This is the cluster duration for Agent-info sync task in the setup phase. As shown in the previous update, 4.8.0 is way slower than 4.7.3 and (also now) 4.7.4:
| v4.7.3 | v4.7.4 | v4.8.0 | ||
|---|---|---|---|---|
| mean | master | 0.088 | 0.152 | 0.687 |
| workers | 0.203 | 0.262 | 0.950 | |
| max | master | 0.914 | 2.33 | 10.257 |
| workers | 1.041 | 2.468 | 10.639 |
|
As described by @Selutario in the previous comment, it can be seen an increase in the times of v4.8.0 compared to v4.7.4, mainly in the Line 366 in bce70d9
wazuh/src/wazuh_db/wdb_parser.c Lines 1177 to 1185 in bce70d9 We are testing to check if the delay is caused by these changes. |
Test 4.8.0-rc1Based on tag 4.8.0-rc1 without any changes. /wdb-query2.py global "sql PRAGMA synchronous"
[
{
"synchronous": 1
}
]
cat /var/ossec/etc/client.keys | wc -l
50000
TOKEN=$(curl -u wazuh:wazuh -k -X POST "https://localhost:55000/security/user/authenticate?raw=true")
curl -k -X POST "https://localhost:55000/groups" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d '{"group_id": "test_group7"}'
{"message": "Group 'test_group7' created.", "error": 0}
time curl -k -X PUT "https://localhost:55000/agents/group?wait_for_complete=true&group_id=test_group7" -H "Authorization: Bearer $TOKEN"
{"data": {"affected_items": ["001","50000"], "total_affected_items": 50000, "total_failed_items": 1, "failed_items": [{"error": {"code": 2003, "message": "Error in wazuhdb request: An error occurred during the set of the groups", "remediation": "Make sure the request is correct"}, "id": ["50001"]}]}, "message": "Some agents were not assigned to test_group7", "error": 2}
real 7m7,813s
user 0m0,029s
sys 0m0,047s
Statistics
Results:
Test 4.8.0-rc1 synchronous = 2Based on tag 4.8.0-rc1 without set New code// Opens global database and stores it in DB pool. It returns a locked database or NULL
wdb_t * wdb_open_global() {
char path[PATH_MAX + 1] = "";
wdb_t * wdb = wdb_pool_get_or_create(WDB_GLOB_NAME);
if (wdb->db == NULL) {
// Try to open DB
snprintf(path, sizeof(path), "%s/%s.db", WDB2_DIR, WDB_GLOB_NAME);
if (sqlite3_open_v2(path, &wdb->db, SQLITE_OPEN_READWRITE, NULL)) {
mdebug1("Global database not found, creating.");
wdb_close(wdb, false);
// Creating database
if (OS_SUCCESS != wdb_create_global(path)) {
merror("Couldn't create SQLite database '%s'", path);
wdb_pool_leave(wdb);
return NULL;
}
// Retry to open
if (sqlite3_open_v2(path, &wdb->db, SQLITE_OPEN_READWRITE, NULL)) {
merror("Can't open SQLite database '%s': %s", path, sqlite3_errmsg(wdb->db));
wdb_close(wdb, false);
wdb_pool_leave(wdb);
return NULL;
}
} else {
if (wdb_upgrade_global(wdb) == NULL || wdb->db == NULL) {
wdb_pool_leave(wdb);
return NULL;
}
}
wdb_enable_foreign_keys(wdb->db);
//wdb_set_synchronous_normal(wdb); <------------- removed
}
return wdb;
}/wdb-query2.py global "sql PRAGMA synchronous"
[
{
"synchronous": 2
}
]
cat /var/ossec/etc/client.keys | wc -l
50000
TOKEN=$(curl -u wazuh:wazuh -k -X POST "https://localhost:55000/security/user/authenticate?raw=true")
curl -k -X POST "https://localhost:55000/groups" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d '{"group_id": "test_group7"}'
{"message": "Group 'test_group7' created.", "error": 0}
time curl -k -X PUT "https://localhost:55000/agents/group?wait_for_complete=true&group_id=test_group7" -H "Authorization: Bearer $TOKEN"
{"data": {"affected_items": ["001","50000"], "total_affected_items": 50000, "total_failed_items": 1, "failed_items": [{"error": {"code": 2003, "message": "Error in wazuhdb request: An error occurred during the set of the groups", "remediation": "Make sure the request is correct"}, "id": ["50001"]}]}, "message": "Some agents were not assigned to test_group7", "error": 2}
real 10m30,385s
user 0m0,021s
sys 0m0,047s
Statistics
Results:
Test 4.8.0-rc1 no rollback in set-agent-groups.Based on tag 4.8.0-rc1 rollback deleted in New code } else if (strcmp(query, "set-agent-groups") == 0) {
w_inc_global_agent_set_agent_groups();
if (!next) {
mdebug1("Global DB Invalid DB query syntax for set-agent-groups.");
mdebug2("Global DB query error near: %s", query);
snprintf(output, OS_MAXSTR + 1, "err Invalid DB query syntax, near '%.32s'", query);
result = OS_INVALID;
} else {
gettimeofday(&begin, 0);
result = wdb_parse_global_set_agent_groups(wdb, next, output);
/*
if (wdb_commit2(wdb) < 0) {
snprintf(output, OS_MAXSTR + 1, "err Cannot commit current transaction to continue");
result = OS_INVALID;
} else {
result = wdb_parse_global_set_agent_groups(wdb, next, output);
if (result == OS_INVALID && wdb_rollback2(wdb) < 0) {
mdebug1("Global DB Cannot rollback transaction.");
}
}
*/
gettimeofday(&end, 0);
timersub(&end, &begin, &diff);
w_inc_global_agent_set_agent_groups_time(diff);
}
} /wdb-query2.py global "sql PRAGMA synchronous"
[
{
"synchronous": 1
}
]
cat /var/ossec/etc/client.keys | wc -l
50000
TOKEN=$(curl -u wazuh:wazuh -k -X POST "https://localhost:55000/security/user/authenticate?raw=true")
curl -k -X POST "https://localhost:55000/groups" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d '{"group_id": "test_group7"}'
{"message": "Group 'test_group7' created.", "error": 0}
time curl -k -X PUT "https://localhost:55000/agents/group?wait_for_complete=true&group_id=test_group7" -H "Authorization: Bearer $TOKEN"
{"data": {"affected_items": ["001","50000"], "total_affected_items": 50000, "total_failed_items": 1, "failed_items": [{"error": {"code": 2003, "message": "Error in wazuhdb request: An error occurred during the set of the groups", "remediation": "Make sure the request is correct"}, "id": ["50001"]}]}, "message": "Some agents were not assigned to test_group7", "error": 2}
real 0m40,348s
user 0m0,017s
sys 0m0,012s
Statistics
Results:
Conclusion
|
Conclusion
Related issuesThese tasks have been introduced in 4.8.0:
RationaleAdding an agent to a group (#16058) entails several intermediate operations that must be performed within a single transaction to allow for rollback. This necessitates opening a transaction for each operation and incurring the associated cost, as we have already opted to avoid bundling all operations into a single transaction. This pertains specifically to the Additionally, we have discovered that the performance of the agent databases improved when changing the disk sync policy (#22774). However, it should be noted that there is a cost associated with changing the policy each time the global database is opened. Furthermore, we have ruled out the option of keeping global.db open indefinitely. |
Requested changeWe need to revert #16058. |
1 task
|
Working branch: |
9 tasks
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
We have detected a degraded wazuh-db performance again (since we already reported something similar in #22629) while running the Workload benchmarks:
test_cluster_performance#23331This is especially noticeable when comparing with previous versions. For example, this was the time it took for clusterd (master) to update 32 chunks in wazuh-db in v4.7.3:
And this is in 4.8.0-rc1:
How to reproduce
client.keyswith this one (50k agents).cluster.logthere should appear logs like this:Additional details
Doing what was described above, both in a 4.7.4 and 4.8.0 environment, it can be seen a clearly worse performance, although not as pronounced as that found in the workload benchmark.
v4.7.4
Average: 0.641s
v4.8.0
Average: Avg: 0.966s (+50.7% compared to 4.7.4)
Also, a slower response in endpoints that require information from wazuh-db (like GET /overview/agents) was found.
The text was updated successfully, but these errors were encountered: