Broken database during Vulnerability Detector tests #22847
Assignees
Labels
level/task
module/db
Wazuh DB engine
type/bug/regression
Breaks functionality known to work in previous releases
type/bug
Something isn't working
Comments
|
The issue has been replicated enabling modulesd debug. This has been carried on this build: https://ci.wazuh.info/job/Test_e2e_system/256/ |
|
Tests were re-run on the following job: https://ci.wazuh.info/job/Test_e2e_system/260, where the errors occurred again. We could verify after the tests were run, that the disk usage was not either full or close to its capacity limit: |
|
At present, attempts to manually replicate the issue have been unsuccessful. We've been unable to reproduce the reported issue in a single manager node local environment. Further research is needed to address this issue comprehensively. |
|
I am trying to reproduce the issue in a local environment, by mounting a cluster. I will update the issue if I find any relevant information. |
|
Tried to reproduce the error manually, but still no success. The error continues to be reproduced consistently in the test. Two methods will continue to be tried:
|
|
I still couldn't reproduce the issue in my local environment. I have a setup with two managers and two agents in a cluster configuration and I am getting different errors while I run the tests, which we are analyzing with @juliamagan and @Rebits. |
|
We are reopening this issue because the error appeared again during the execution of the Vulnerability Detector tests in Beta 6 (#23060). |
|
Possibly related to #18955. We need to block this issue until we receive the steps to reproduce the problem locally, or access to an environment where the problem can be reproduced. |
Closed
1 task
|
This issue is blocked until we obtain information from: |
|
As soon as I get access to both vms (worker and manager) , I wasn't able to find the error messages, (the present ossec.log didn't show them and there was no older logs available) Additional checks from my side: Master 0No empty DB found: root@ip-172-31-0-22:/var/ossec/queue# cd db/
root@ip-172-31-0-22:/var/ossec/queue/db# ls -l
total 6208
-rw-r----- 1 wazuh wazuh 3624960 Apr 30 17:45 000.db
-rw-r----- 1 wazuh wazuh 245760 Apr 30 17:44 001.db
-rw-r----- 1 wazuh wazuh 409600 Apr 30 18:41 002.db
-rw-r----- 1 wazuh wazuh 25136 Apr 30 18:41 002.db-journal
-rw-r----- 1 wazuh wazuh 245760 Apr 30 17:44 003.db
-rw-r----- 1 wazuh wazuh 512000 Apr 30 18:42 004.db
-rw-r----- 1 wazuh wazuh 25136 Apr 30 18:42 004.db-journal
-rw-r----- 1 wazuh wazuh 716800 Apr 30 18:42 005.db
-rw-r----- 1 wazuh wazuh 25136 Apr 30 18:42 005.db-journal
-rw-r----- 1 wazuh wazuh 385024 Apr 30 18:41 006.db
-rw-r----- 1 wazuh wazuh 16928 Apr 30 18:41 006.db-journal
-rw-r----- 1 wazuh wazuh 81920 Apr 30 18:41 global.db
-rw-r----- 1 wazuh wazuh 21032 Apr 30 18:42 global.db-journal
srw-rw---- 1 wazuh wazuh 0 Apr 30 17:43 wdbNo error messages related to wdb: root@ip-172-31-0-22:/var/ossec# cat logs/ossec.log | grep "sqlite3_prepare_v2"
root@ip-172-31-0-22:/var/ossec# cat logs/ossec.log | grep "wazuh-db" | grep "ERROR"
root@ip-172-31-0-22:/var/ossec# Cheking DBs state with manager stopped or running always throw the same result: root@ip-172-31-0-22:/var/ossec/queue/db# sqlite3 global.db
SQLite version 3.37.2 2022-01-06 13:25:41
Enter ".help" for usage hints.
sqlite> PRAGMA integrity_check ;
ok
sqlite> .quit
root@ip-172-31-0-22:/var/ossec/queue/db# sqlite3 000.db
SQLite version 3.37.2 2022-01-06 13:25:41
Enter ".help" for usage hints.
sqlite> PRAGMA integrity_check ;
ok
sqlite> .quit
root@ip-172-31-0-22:/var/ossec/queue/db# sqlite3 001.db
SQLite version 3.37.2 2022-01-06 13:25:41
Enter ".help" for usage hints.
sqlite> PRAGMA integrity_check ;
ok
sqlite> .quit
root@ip-172-31-0-22:/var/ossec/queue/db# sqlite3 002.db
SQLite version 3.37.2 2022-01-06 13:25:41
Enter ".help" for usage hints.
sqlite> PRAGMA integrity_check ;
ok
sqlite> .quit
root@ip-172-31-0-22:/var/ossec/queue/db# sqlite3 003.db
SQLite version 3.37.2 2022-01-06 13:25:41
Enter ".help" for usage hints.
sqlite> PRAGMA integrity_check ;
ok
sqlite> .quit
root@ip-172-31-0-22:/var/ossec/queue/db# sqlite3 004.db
SQLite version 3.37.2 2022-01-06 13:25:41
Enter ".help" for usage hints.
sqlite> PRAGMA integrity_check ;
ok
sqlite> .quit
root@ip-172-31-0-22:/var/ossec/queue/db# sqlite3 005.db
SQLite version 3.37.2 2022-01-06 13:25:41
Enter ".help" for usage hints.
sqlite> PRAGMA integrity_check ;
ok
sqlite> .quit
root@ip-172-31-0-22:/var/ossec/queue/db# sqlite3 006.db
SQLite version 3.37.2 2022-01-06 13:25:41
Enter ".help" for usage hints.
sqlite> PRAGMA integrity_check ;
ok
sqlite> .quitWorker 0There's no empty DB root@ip-172-31-14-223:/home/qa# cd /var/ossec/queue/db/
root@ip-172-31-14-223:/var/ossec/queue/db# ls -l
total 41076
-rw-r----- 1 wazuh wazuh 3571712 Apr 30 17:45 000.db
-rw-r----- 1 wazuh wazuh 37773312 Apr 30 18:42 001.db
-rw-r----- 1 wazuh wazuh 21032 Apr 30 18:42 001.db-journal
-rw-r----- 1 wazuh wazuh 573440 Apr 30 18:42 003.db
-rw-r----- 1 wazuh wazuh 25136 Apr 30 18:42 003.db-journal
-rw-r----- 1 wazuh wazuh 81920 Apr 30 18:43 global.db
srw-rw---- 1 wazuh wazuh 0 Apr 30 17:43 wdbNo error logs related to wdb or the base error message root@ip-172-31-14-223:/var/ossec/logs# cat ossec.log | grep "wazuh-db" | grep "ERROR"
root@ip-172-31-14-223:/var/ossec/logs# cat ossec.log | grep "sqlite3_prepare_v2"
root@ip-172-31-14-223:/var/ossec/logs# DB status throws the same result, for manager running or stopped: root@ip-172-31-14-223:/var/ossec/queue/db# sqlite3 global.db
sqlite> PRAGMA integrity_check ;
ok
root@ip-172-31-14-223:/var/ossec/queue/db# sqlite3 000.db
sqlite> PRAGMA integrity_check ;
ok
root@ip-172-31-14-223:/var/ossec/queue/db# sqlite3 001.db
sqlite> PRAGMA integrity_check ;
ok
root@ip-172-31-14-223:/var/ossec/queue/db# sqlite3 002.db
sqlite> PRAGMA integrity_check ;
ok
root@ip-172-31-14-223:/var/ossec/queue/db# sqlite3 003.db
sqlite> PRAGMA integrity_check ;
ok
root@ip-172-31-14-223:/var/ossec/queue/db# sqlite3 004.db
sqlite> PRAGMA integrity_check ;
ok
root@ip-172-31-14-223:/var/ossec/queue/db# sqlite3 005.db
sqlite> PRAGMA integrity_check ;
ok
root@ip-172-31-14-223:/var/ossec/queue/db# sqlite3 006.db
sqlite> PRAGMA integrity_check ;
ok |
|
This issue appears again during Release 4.8.0 - RC 1 - Vulnerability Detection E2E tests Report: https://github.com/wazuh/wazuh/files/15234960/Test_e2e_system_284_test_vulnerability_detector.zip |
|
Reopening wazuh/wazuh-qa#5309. Update: unblocked. |
Closed
DiagnosticsLogsossec.log | wazuh-dbStartup / shutdownRecreated databasesError sampleDatabase files statsBroken databaseSQL schemaCREATE TABLE fim_entry ( full_path TEXT NOT NULL PRIMARY KEY, file TEXT, type TEXT NOT NULL CHECK (type IN ('file', 'registry_key', 'registry_value')), date INTEGER NOT NULL DEFAULT (strftime('%s', 'now')), changes INTEGER NOT NULL DEFAULT 1, arch TEXT CHECK (arch IN (NULL, '[x64]', '[x32]')), value_name TEXT, value_type TEXT, size INTEGER, perm TEXT, uid TEXT, gid TEXT, md5 TEXT, sha1 TEXT, uname TEXT, gname TEXT, mtime INTEGER, inode INTEGER, sha256 TEXT, attributes TEXT, symbolic_path TEXT, checksum TEXT);
CREATE INDEX fim_full_path_index ON fim_entry (full_path);
CREATE INDEX fim_file_index ON fim_entry (file);
CREATE INDEX fim_date_index ON fim_entry (date);
CREATE INDEX fim_type_full_path_index ON fim_entry (type, full_path);
CREATE INDEX fim_type_file_index ON fim_entry (type, file);
CREATE INDEX fim_type_full_path_checksum_index ON fim_entry(type,full_path,checksum);
CREATE INDEX fim_type_file_checksum_index ON fim_entry(type,file,checksum);
CREATE TABLE pm_event ( id INTEGER PRIMARY KEY AUTOINCREMENT, date_first INTEGER, date_last INTEGER, log TEXT, pci_dss TEXT, cis TEXT);
CREATE TABLE sqlite_sequence(name,seq);
CREATE INDEX pm_event_log ON pm_event (log);
CREATE INDEX pm_event_date ON pm_event (date_last);
CREATE TABLE sys_netiface ( scan_id INTEGER, scan_time TEXT, name TEXT, adapter TEXT, type TEXT, state TEXT, mtu INTEGER CHECK (mtu > 0), mac TEXT, tx_packets INTEGER, rx_packets INTEGER, tx_bytes INTEGER, rx_bytes INTEGER, tx_errors INTEGER, rx_errors INTEGER, tx_dropped INTEGER, rx_dropped INTEGER, checksum TEXT NOT NULL CHECK (checksum <> ''), item_id TEXT, PRIMARY KEY (scan_id, name));
CREATE INDEX netiface_id ON sys_netiface (scan_id);
CREATE TABLE sys_netproto ( scan_id INTEGER REFERENCES sys_netiface (scan_id), iface TEXT REFERENCES sys_netiface (name), type TEXT, gateway TEXT, dhcp TEXT NOT NULL CHECK (dhcp IN ('enabled', 'disabled', 'unknown', 'BOOTP')) DEFAULT 'unknown', metric INTEGER, checksum TEXT NOT NULL CHECK (checksum <> ''), item_id TEXT, PRIMARY KEY (scan_id, iface, type));
CREATE INDEX netproto_id ON sys_netproto (scan_id);
CREATE TABLE sys_netaddr ( scan_id INTEGER REFERENCES sys_netproto (scan_id), iface TEXT REFERENCES sys_netproto (iface), proto TEXT REFERENCES sys_netproto (type), address TEXT, netmask TEXT, broadcast TEXT, checksum TEXT NOT NULL CHECK (checksum <> ''), item_id TEXT, PRIMARY KEY (scan_id, iface, proto, address));
CREATE INDEX netaddr_id ON sys_netaddr (scan_id);
CREATE TABLE sys_osinfo ( scan_id INTEGER, scan_time TEXT, hostname TEXT, architecture TEXT, os_name TEXT, os_version TEXT, os_codename TEXT, os_major TEXT, os_minor TEXT, os_patch TEXT, os_build TEXT, os_platform TEXT, sysname TEXT, release TEXT, version TEXT, os_release TEXT, checksum TEXT NOT NULL CHECK (checksum <> ''), os_display_version TEXT, reference TEXT NOT NULL DEFAULT '', PRIMARY KEY (scan_id, os_name));
CREATE TABLE sys_hwinfo ( scan_id INTEGER, scan_time TEXT, board_serial TEXT, cpu_name TEXT, cpu_cores INTEGER, cpu_mhz REAL, ram_total INTEGER, ram_free INTEGER, ram_usage INTEGER, checksum TEXT NOT NULL CHECK (checksum <> ''), PRIMARY KEY (scan_id, board_serial));
CREATE TABLE sys_ports ( scan_id INTEGER, scan_time TEXT, protocol TEXT, local_ip TEXT, local_port INTEGER CHECK (local_port >= 0), remote_ip TEXT, remote_port INTEGER CHECK (remote_port >= 0), tx_queue INTEGER, rx_queue INTEGER, inode INTEGER, state TEXT, PID INTEGER, process TEXT, checksum TEXT NOT NULL CHECK (checksum <> ''), item_id TEXT, PRIMARY KEY (protocol, local_ip, local_port, inode));
CREATE INDEX ports_id ON sys_ports (scan_id);
CREATE TABLE sys_programs ( scan_id INTEGER, scan_time TEXT, format TEXT, name TEXT, priority TEXT, section TEXT, size INTEGER CHECK (size >= 0), vendor TEXT, install_time TEXT, version TEXT, architecture TEXT, multiarch TEXT, source TEXT, description TEXT, location TEXT, cpe TEXT, msu_name TEXT, checksum TEXT NOT NULL CHECK (checksum <> ''), item_id TEXT, PRIMARY KEY (scan_id, name, version, architecture, format, location));
CREATE INDEX programs_id ON sys_programs (scan_id);
CREATE TABLE sys_hotfixes ( scan_id INTEGER, scan_time TEXT, hotfix TEXT, checksum TEXT NOT NULL CHECK (checksum <> ''), PRIMARY KEY (scan_id, hotfix));
CREATE INDEX hotfix_id ON sys_hotfixes (scan_id);
CREATE TABLE sys_processes ( scan_id INTEGER, scan_time TEXT, pid TEXT, name TEXT, state TEXT, ppid INTEGER, utime INTEGER, stime INTEGER, cmd TEXT, argvs TEXT, euser TEXT, ruser TEXT, suser TEXT, egroup TEXT, rgroup TEXT, sgroup TEXT, fgroup TEXT, priority INTEGER, nice INTEGER, size INTEGER, vm_size INTEGER, resident INTEGER, share INTEGER, start_time INTEGER, pgrp INTEGER, session INTEGER, nlwp INTEGER, tgid INTEGER, tty INTEGER, processor INTEGER, checksum TEXT NOT NULL CHECK (checksum <> ''), PRIMARY KEY (scan_id, pid));
CREATE INDEX processes_id ON sys_processes (scan_id);
CREATE TABLE ciscat_results ( id INTEGER PRIMARY KEY AUTOINCREMENT, scan_id INTEGER, scan_time TEXT, benchmark TEXT, profile TEXT, pass INTEGER, fail INTEGER, error INTEGER, notchecked INTEGER, unknown INTEGER, score INTEGER);
CREATE INDEX ciscat_id ON ciscat_results (scan_id);
CREATE TABLE metadata ( key TEXT PRIMARY KEY, value TEXT);
CREATE TABLE scan_info ( module TEXT PRIMARY KEY, first_start INTEGER DEFAULT 0, first_end INTEGER DEFAULT 0, start_scan INTEGER DEFAULT 0, end_scan INTEGER DEFAULT 0, fim_first_check INTEGER DEFAULT 0, fim_second_check INTEGER DEFAULT 0, fim_third_check INTEGER DEFAULT 0);
CREATE TABLE sca_policy ( name TEXT, file TEXT, id TEXT, description TEXT, `references` TEXT, hash_file TEXT);
CREATE TABLE sca_scan_info ( id INTEGER PRIMARY KEY, start_scan INTEGER, end_scan INTEGER, policy_id TEXT REFERENCES sca_policy (id), pass INTEGER, fail INTEGER, invalid INTEGER, total_checks INTEGER, score INTEGER, hash TEXT);
CREATE TABLE sca_check ( scan_id INTEGER REFERENCES sca_scan_info (id), id INTEGER PRIMARY KEY, policy_id TEXT REFERENCES sca_policy (id), title TEXT, description TEXT, rationale TEXT, remediation TEXT, file TEXT, process TEXT, directory TEXT, registry TEXT, command TEXT, `references` TEXT, result TEXT, reason TEXT, condition TEXT);
CREATE INDEX policy_id_index ON sca_check (policy_id);
CREATE TABLE sca_check_rules ( id_check INTEGER REFERENCES sca_check (id), `type` TEXT, rule TEXT, PRIMARY KEY (id_check, `type`, rule));
CREATE INDEX rules_id_check_index ON sca_check_rules (id_check);
CREATE TABLE sca_check_compliance ( id_check INTEGER REFERENCES sca_check (id), `key` TEXT, `value` TEXT, PRIMARY KEY (id_check, `key`, `value`));
CREATE INDEX comp_id_check_index ON sca_check_compliance (id_check);Note: Agent 002 backup file was lost. Difference from what was expected41d40
< CREATE TABLE sync_info ( component TEXT PRIMARY KEY, last_attempt INTEGER DEFAULT 0, last_completion INTEGER DEFAULT 0, n_attempts INTEGER DEFAULT 0, n_completions INTEGER DEFAULT 0, last_manager_checksum TEXT NOT NULL DEFAULT '', last_agent_checksum TEXT NOT NULL DEFAULT '');Agent list
ArtifactsConclusions
We haven't found the cause of this issue yet. |
|
With some help from @juliamagan I was able to execute the whole end_to_end test_vulnerability_detector in the created environment. While it was being executed I was checking the ossec logs from both managers if the error came up but sadly it didn't. python3 -m pytest --pdb tests/end_to_end/test_vulnerability_detector/ --gather-evidences-when-passed --enable-verbose-evidences --inventory-path=/home/vagrant/E2EVulnerability/Wazuh_QA_environment1154_testing_inventory.yaml --html=replicate_database_error2.htmlThe result of it (after 2 hours) was: This is where I have some doubt because it just shows 1 result while in the proccess I was able to see many other errors from the previous ones. I'll continue digging into the code and retrying the tests in order to replicate the main error. |
|
@LucioDonda, initial tests are failing right at the outset, even before establishing connections with remote hosts. The host manager object represents the handler for remote hosts, contingent upon a valid Ansible inventory. The report indicates a failure during the validation step: If we check test duration we can see that this was aborted before it started: |
|
I was able to reproduce the issue. after the environment was re-built I executed a smaller version (taking less time than the whole set): In this case, the DBs envolvs where 002.db and 004.db both linux agents: root@ip-172-31-7-25:/var/ossec# bin/manage_agents -l
Available agents:
ID: 001, Name: agent4, IP: any
ID: 002, Name: agent1, IP: any <---
ID: 003, Name: agent5, IP: any
ID: 004, Name: agent6, IP: any <---
ID: 005, Name: agent3, IP: anysqlite> SELECT * FROM agent ;
id|name|ip|register_ip|internal_key|os_name|os_version|os_major|os_minor|os_codename|os_build|os_platform|os_uname|os_arch|version|config_sum|merged_sum|manager_host|node_name|date_add|last_keepalive|group|group_hash|group_sync_status|sync_status|connection_status|disconnection_time|group_config_status|status_code
2|agent1|172.31.4.190|any|72670cced442c79c66becbc8d70f79c047025e9411289e75aaea3546b9cbbde1|CentOS Linux|7.9|7|9|||centos|Linux |ip-172-31-4-190.ec2.internal |3.10.0-1160.102.1.el7.x86_64 |#1 SMP Tue Oct 17 15:42:21 UTC 2023 |x86_64|x86_64|Wazuh v4.8.0|ab73af41699f13fdd81903b5f23d8d00|4a8724b20dee0124ff9656783c490c4e|ip-172-31-7-25|master|1715359497|1715371019|default|37a8eec1|synced|synced|active|0|synced|0
4|agent6|172.31.13.14|any|171d3184ac3f6d101acaaaebc236fae547c543db7d5b984be82905644ac49a77|Ubuntu|22.04.3 LTS|22|04|Jammy Jellyfish||ubuntu|Linux |ip-172-31-13-14 |6.2.0-1015-aws |#15~22.04.1-Ubuntu SMP Fri Oct 6 23:21:18 UTC 2023 |aarch64|aarch64|Wazuh v4.8.0|ab73af41699f13fdd81903b5f23d8d00|4a8724b20dee0124ff9656783c490c4e|ip-172-31-7-25|master|1715359497|1715371019|default|37a8eec1|synced|synced|active|0|synced|0Report result of the test: Example of logs in the manager: 2024/05/10 18:17:42 wazuh-modulesd:vulnerability-scanner[81928] scanOrchestrator.hpp:282 at run(): DEBUG: Event type: 11 processed
2024/05/10 18:17:42 indexer-connector[81928] indexerConnector.cpp:129 at abuseControl(): DEBUG: Agent '001' sync ommited due to abuse control.
2024/05/10 18:17:42 wazuh-modulesd:vulnerability-scanner[81928] scanOrchestrator.hpp:282 at run(): DEBUG: Event type: 11 processed
2024/05/10 18:17:42 indexer-connector[81928] indexerConnector.cpp:129 at abuseControl(): DEBUG: Agent '001' sync ommited due to abuse control.
2024/05/10 18:17:54 wazuh-modulesd:vulnerability-scanner[81928] osScanner.hpp:340 at handleRequest(): DEBUG: Vulnerability scan for OS 'enterprise_linux' on Agent '002' has completed.
2024/05/10 18:17:54 wazuh-modulesd:vulnerability-scanner[81928] eventDetailsBuilder.hpp:101 at handleRequest(): DEBUG: Building event details for component type: 2
2024/05/10 18:17:54 wazuh-modulesd:vulnerability-scanner[81928] scanOrchestrator.hpp:282 at run(): DEBUG: Event type: 3 processed
2024/05/10 18:17:55 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
2024/05/10 18:17:55 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/10 18:17:55 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processesIn this case, the DBs are not empty nor broken, but some of the table are missing: DBs for analysis: After that, I tried to replicate the issue recovering one of them and leaving the other one broken, but I wasn't able to achieve it. There's some propcedure on the first moments of the manager (or the whole environment) where those first test triggers this issue Additional Information: daemons : # curl -k -X GET "https://localhost:55000/manager/daemons/stats?daemons_list=wazuh-db,wazuh-remoted{
"data": {
"affected_items": [
{
"metrics": {
"queries": {
"received": 18881,
"received_breakdown": {
"agent": 8826,
"agent_breakdown": {
"db": {
"begin": 0,
"close": 0,
"commit": 0,
"get_fragmentation": 0,
"remove": 0,
"sleep": 0,
"sql": 0,
"vacuum": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 0
},
"sca": {
"sca": 0
},
"sync": {
"dbsync": 5184
},
"syscheck": {
"fim_file": 0,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 0
},
"syscollector": {
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
},
"syscollector_hotfixes": 343,
"syscollector_hwinfo": 601,
"syscollector_network_address": 343,
"syscollector_network_iface": 346,
"syscollector_network_protocol": 343,
"syscollector_osinfo": 344,
"syscollector_packages": 343,
"syscollector_ports": 358,
"syscollector_processes": 621
}
}
},
"global": 8646,
"global_breakdown": {
"db": {
"backup": 0,
"get_fragmentation": 0,
"sleep": 0,
"sql": 0,
"vacuum": 0
},
"tables": {
"agent": {
"delete-agent": 0,
"disconnect-agents": 265,
"find-agent": 0,
"get-agent-info": 0,
"get-agents-by-connection-status": 2126,
"get-all-agents": 1,
"get-distinct-groups": 532,
"get-groups-integrity": 0,
"insert-agent": 0,
"reset-agents-connection": 1,
"select-agent-group": 2128,
"select-agent-name": 8,
"set-agent-groups": 0,
"sync-agent-groups-get": 529,
"sync-agent-info-get": 0,
"sync-agent-info-set": 530,
"update-agent-data": 2129,
"update-agent-name": 1,
"update-connection-status": 0,
"update-keepalive": 4,
"update-status-code": 0
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 0,
"find-group": 1,
"insert-agent-group": 0,
"select-groups": 1
},
"labels": {
"get-labels": 390
}
}
},
"mitre": 1400,
"mitre_breakdown": {
"db": {
"sql": 1400
}
},
"task": 9,
"task_breakdown": {
"db": {
"sql": 0
},
"tables": {
"tasks": {
"delete_old": 1,
"set_timeout": 6,
"upgrade": 0,
"upgrade_cancel_tasks": 2,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 0,
"wazuhdb_breakdown": {
"db": {
"remove": 0
}
}
}
},
"time": {
"execution": 21154,
"execution_breakdown": {
"agent": 20556,
"agent_breakdown": {
"db": {
"begin": 0,
"close": 0,
"commit": 0,
"get_fragmentation": 0,
"open": 102,
"remove": 0,
"sleep": 0,
"sql": 0,
"vacuum": 0
},
"tables": {
"ciscat": {
"ciscat": 0
},
"rootcheck": {
"rootcheck": 0
},
"sca": {
"sca": 0
},
"sync": {
"dbsync": 20103
},
"syscheck": {
"fim_file": 0,
"fim_registry": 0,
"fim_registry_key": 0,
"fim_registry_value": 0,
"syscheck": 0
},
"syscollector": {
"deprecated": {
"hardware": 0,
"hotfix": 0,
"netaddr": 0,
"netinfo": 0,
"netproto": 0,
"osinfo": 0,
"package": 0,
"port": 0,
"process": 0
},
"syscollector_hotfixes": 12,
"syscollector_hwinfo": 67,
"syscollector_network_address": 22,
"syscollector_network_iface": 44,
"syscollector_network_protocol": 35,
"syscollector_osinfo": 28,
"syscollector_packages": 66,
"syscollector_ports": 23,
"syscollector_processes": 49
}
}
},
"global": 517,
"global_breakdown": {
"db": {
"backup": 0,
"get_fragmentation": 0,
"open": 20,
"sleep": 0,
"sql": 0,
"vacuum": 0
},
"tables": {
"agent": {
"delete-agent": 0,
"disconnect-agents": 6,
"find-agent": 0,
"get-agent-info": 0,
"get-agents-by-connection-status": 99,
"get-all-agents": 0,
"get-distinct-groups": 26,
"get-groups-integrity": 0,
"insert-agent": 0,
"reset-agents-connection": 0,
"select-agent-group": 51,
"select-agent-name": 0,
"set-agent-groups": 0,
"sync-agent-groups-get": 19,
"sync-agent-info-get": 0,
"sync-agent-info-set": 74,
"update-agent-data": 201,
"update-agent-name": 0,
"update-connection-status": 0,
"update-keepalive": 0,
"update-status-code": 0
},
"belongs": {
"get-group-agents": 0,
"select-group-belong": 0
},
"group": {
"delete-group": 0,
"find-group": 1,
"insert-agent-group": 0,
"select-groups": 0
},
"labels": {
"get-labels": 15
}
}
},
"mitre": 81,
"mitre_breakdown": {
"db": {
"sql": 81
}
},
"task": 0,
"task_breakdown": {
"db": {
"sql": 0
},
"tables": {
"tasks": {
"delete_old": 0,
"set_timeout": 0,
"upgrade": 0,
"upgrade_cancel_tasks": 0,
"upgrade_custom": 0,
"upgrade_get_status": 0,
"upgrade_result": 0,
"upgrade_update_status": 0
}
}
},
"wazuhdb": 0,
"wazuhdb_breakdown": {
"db": {
"remove": 0
}
}
}
}
},
"name": "wazuh-db",
"timestamp": "2024-05-10T19:49:37+00:00",
"uptime": "2024-05-10T18:20:53+00:00"
}
],
"failed_items": [],
"total_affected_items": 1,
"total_failed_items": 0
},
"error": 0,
"message": "Statistical information for each daemon was successfully read"
}
agents: # curl -k -X GET "https://localhost:55000/overview/agents"{
"data": {
"agent_os": [
{
"count": 2,
"os": {
"name": "CentOS Linux",
"platform": "centos",
"version": "7.9"
}
},
{
"count": 2,
"os": {
"name": "Ubuntu",
"platform": "ubuntu",
"version": "22.04.3 LTS"
}
},
{
"count": 1,
"os": {
"name": "Ubuntu",
"platform": "ubuntu",
"version": "22.04 LTS"
}
},
{
"count": 1,
"os": {
"name": "Microsoft Windows 11 Pro",
"platform": "windows",
"version": "10.0.22621.1105"
}
}
],
"agent_status": {
"configuration": {
"not_synced": 0,
"synced": 6,
"total": 6
},
"connection": {
"active": 6,
"disconnected": 0,
"never_connected": 0,
"pending": 0,
"total": 6
}
},
"agent_version": [
{
"count": 6,
"version": "Wazuh v4.8.0"
}
],
"groups": [
{
"configSum": "ab73af41699f13fdd81903b5f23d8d00",
"count": 6,
"mergedSum": "4a8724b20dee0124ff9656783c490c4e",
"name": "default"
}
],
"last_registered_agent": [
{
"configSum": "ab73af41699f13fdd81903b5f23d8d00",
"dateAdd": "2024-05-10T16:45:01+00:00",
"group": [
"default"
],
"group_config_status": "synced",
"id": "006",
"ip": "172.31.1.80",
"lastKeepAlive": "2024-05-10T22:28:38+00:00",
"manager": "ip-172-31-11-24",
"mergedSum": "4a8724b20dee0124ff9656783c490c4e",
"name": "agent2",
"node_name": "manager2",
"os": {
"build": "22621.1105",
"major": "10",
"minor": "0",
"name": "Microsoft Windows 11 Pro",
"platform": "windows",
"uname": "Microsoft Windows 11 Pro",
"version": "10.0.22621.1105"
},
"registerIP": "any",
"status": "active",
"status_code": 0,
"version": "Wazuh v4.8.0"
}
],
"nodes": [
{
"count": 4,
"node_name": "master"
},
{
"count": 2,
"node_name": "manager2"
}
]
},
"error": 0
}
|
|
I wasn't able to recreate the error again due to some problems accesing the environment. By checking the logs, in the first test case where the issue was found, before the test itself starts, it can be seen some error messages on the logs: 1st error message ( 2024/05/07 09:07:35 ) In the other case where we were able to replicate the issue this doesn't repeats, but there's a strange timestamp difference, all the agents and managers holds a similar time, but the test lays in a very different point: Based on those comments I'm having several doubts regarding the way the environment gets up and running, can this be causing the issue? In the meantime I analyzed some changes in the code that could have affected the DB creation, one of them is the addition of a new schema, but is hard to say that it can have some impact. |
|
As discussed recently we have decided to elaborate a script that could monitor the DB status. Based on that I've been developing a script but I wasn't able to finish yet. Pending features:
In order to check what was mentioned on the previous comment this script should be executed in the manager (master) before installing all the wazuh components, and it should remain running till the test is finished. Script WIP:
import subprocess
import sys
import re
# list db command
cmd_list_dbs = "ls -l /var/ossec/queue/db/ | grep .db"
# Execute it
p = subprocess.Popen(cmd_list_dbs, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
# Wait for the command to finish and capture the output
stdout, stderr = p.communicate()
result = ""
# Check if the command executed successfully
if p.returncode == 0:
# Command executed successfully
#print(stdout.decode(sys.stdout.encoding))
result = stdout.decode(sys.stdout.encoding)
else:
print("Command failed with exit code", p.returncode)
print(stderr.decode(sys.stderr.encoding))
dbs_list = []
if result:
for line in result.splitlines():
m = re.match('.*\s(\d*\.db)', line)
if m:
#print(m[1])
dbs_list.append(m[1])
cmd_tables_dbs = "sqlite3 /var/ossec/queue/db/{} '.tables'"
# Checking each table from each db
if dbs_list:
for db_name in dbs_list:
p = subprocess.Popen(cmd_tables_dbs.format(db_name), shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
tables = ""
if p.returncode == 0:
#Advise when db with no tables
if stdout == "":
tables = "NONE"
else:
tables = stdout.decode(sys.stdout.encoding)
else:
print("Command failed with exit code", p.returncode)
print(stderr.decode(sys.stderr.encoding)) |
Issue reproducedHow to reproduce
Docker imagedocker build -t wazuh-agent:4.7.4 .
for i in {1..50}; do docker run -d wazuh-agent:4.7.4; doneDockerfileFROM rockylinux:9-minimal
ARG WAZUH_MANAGER="172.28.54.78"
ARG WAZUH_VERSION="4.7.4"
ADD wazuh.repo /etc/yum.repos.d/wazuh.repo
ADD run.sh /usr/local/bin/run
RUN microdnf install -y procps wazuh-agent`[ -n "$WAZUH_VERSION" ] && echo "-$WAZUH_VERSION-1"` && \
microdnf clean all
CMD runrun.sh#! /bin/bash
/var/ossec/bin/wazuh-control start
tail -f /var/ossec/logs/ossec.logwazuh.repo[wazuh]
gpgcheck=1
gpgkey=https://packages.wazuh.com/key/GPG-KEY-WAZUH
enabled=1
name=EL-\$releasever - Wazuh
baseurl=https://packages.wazuh.com/4.x/yum/
protect=1Thread sanitizingBuildmake -C src TARGET=server DEBUG=1
./install.sh
cd src
git apply <<\EOF
diff --git a/src/wazuh_db/main.c b/src/wazuh_db/main.c
index 025db17c..db018d88 100644
--- a/src/wazuh_db/main.c
+++ b/src/wazuh_db/main.c
@@ -161,10 +161,6 @@ int main(int argc, char ** argv)
// Change root
- if (Privsep_Chroot(home_path) < 0) {
- merror_exit(CHROOT_ERROR, home_path, errno, strerror(errno));
- }
-
if (Privsep_SetUser(uid) < 0) {
merror_exit(SETUID_ERROR, USER, errno, strerror(errno));
}
EOF
rm wazuh_db/*.o
make TARGET=server DEBUG=1 CFLAGS="-fsanitize=thread -fno-omit-frame-pointer" LDFLAGS="-fsanitize=thread -fno-omit-frame-pointer" wazuh-db
cp wazuh-db /var/ossec/bin/wazuh-db-tsRunpkill -f wazuh-db
/var/ossec/bin/wazuh-db-ts -fThreadSanitizer reportFull reportConclusionThis issue is definitely caused by a race condition of still unknown origin. Next stepsWe need to find out:
|
Test on v4.7.2 🟢Following the steps to reproduce the issue, I have found that it does not occur in v4.7.2. The expected error messages are shown due to wazuh-db stops, but after restarting it the databases are restored correctly. ossec.log2024/05/15 19:50:52 wazuh-csyslogd: INFO: Remote syslog server not configured. Clean exit.
2024/05/15 19:50:52 wazuh-dbd: INFO: Database not configured. Clean exit.
2024/05/15 19:50:52 wazuh-integratord: INFO: Remote integrations not configured. Clean exit.
2024/05/15 19:50:52 wazuh-agentlessd: INFO: Not configured. Exiting.
2024/05/15 19:50:52 wazuh-authd: INFO: Started (pid: 143137).
2024/05/15 19:50:52 wazuh-authd: INFO: Accepting connections on port 1515. No password required.
2024/05/15 19:50:52 wazuh-authd: ERROR: (1401): Error reading authentication key: '
'.
2024/05/15 19:50:52 wazuh-authd: INFO: Setting network timeout to 1.000000 sec.
2024/05/15 19:50:53 wazuh-db: INFO: Started (pid: 143153).
2024/05/15 19:50:54 wazuh-execd: INFO: Started (pid: 143177).
2024/05/15 19:50:56 wazuh-analysisd: INFO: Total rules enabled: '6775'
2024/05/15 19:50:56 wazuh-analysisd: INFO: Started (pid: 143191).
2024/05/15 19:50:56 wazuh-analysisd: INFO: EPS limit disabled
2024/05/15 19:50:56 wazuh-analysisd: INFO: (7200): Logtest started
2024/05/15 19:50:56 wazuh-syscheckd: INFO: Started (pid: 143292).
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6003): Monitoring path: '/bin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'.
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6003): Monitoring path: '/boot', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'.
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6003): Monitoring path: '/etc', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'.
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6003): Monitoring path: '/sbin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'.
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6003): Monitoring path: '/usr/bin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'.
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6003): Monitoring path: '/usr/sbin', with options 'size | permissions | owner | group | mtime | inode | hash_md5 | hash_sha1 | hash_sha256 | scheduled'.
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/mtab'
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/hosts.deny'
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/mail/statistics'
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/random-seed'
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/random.seed'
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/adjtime'
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/httpd/logs'
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/utmpx'
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/wtmpx'
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/cups/certs'
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/dumpdates'
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6206): Ignore 'file' entry '/etc/svc/volatile'
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6207): Ignore 'file' sregex '.log$|.swp$'
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6004): No diff for file: '/etc/ssl/private.key'
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6000): Starting daemon...
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6010): File integrity monitoring scan frequency: 43200 seconds
2024/05/15 19:50:56 wazuh-syscheckd: INFO: (6008): File integrity monitoring scan started.
2024/05/15 19:50:56 rootcheck: INFO: Starting rootcheck scan.
2024/05/15 19:50:57 wazuh-remoted: INFO: Started (pid: 143309). Listening on port 1514/TCP (secure).
2024/05/15 19:50:57 wazuh-remoted: INFO: (1410): Reading authentication keys file.
2024/05/15 19:50:57 wazuh-remoted: ERROR: (1401): Error reading authentication key: '
'.
2024/05/15 19:50:58 wazuh-logcollector: INFO: Monitoring output of command(360): df -P
2024/05/15 19:50:58 wazuh-logcollector: INFO: Monitoring full output of command(360): netstat -tulpn | sed 's/\([[:alnum:]]\+\)\ \+[[:digit:]]\+\ \+[[:digit:]]\+\ \+\(.*\):\([[:digit:]]*\)\ \+\([0-9\.\:\*]\+\).\+\ \([[:digit:]]*\/[[:alnum:]\-]*\).*/\1 \2 == \3 == \4 \5/' | sort -k 4 -g | sed 's/ == \(.*\) ==/:\1/' | sed 1,2d
2024/05/15 19:50:58 wazuh-logcollector: INFO: Monitoring full output of command(360): last -n 20
2024/05/15 19:50:58 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/ossec/logs/active-responses.log'.
2024/05/15 19:50:58 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/log/auth.log'.
2024/05/15 19:50:58 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/log/syslog'.
2024/05/15 19:50:58 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/log/dpkg.log'.
2024/05/15 19:50:58 wazuh-logcollector: INFO: (1950): Analyzing file: '/var/log/kern.log'.
2024/05/15 19:50:58 wazuh-logcollector: INFO: Started (pid: 143377).
2024/05/15 19:50:59 wazuh-syscheckd: INFO: (6009): File integrity monitoring scan ended.
2024/05/15 19:50:59 wazuh-syscheckd: INFO: FIM sync module started.
2024/05/15 19:50:59 wazuh-monitord: INFO: Started (pid: 143435).
2024/05/15 19:51:00 wazuh-modulesd: INFO: Started (pid: 143489).
2024/05/15 19:51:00 wazuh-modulesd:agent-upgrade: INFO: (8153): Module Agent Upgrade started.
2024/05/15 19:51:00 wazuh-modulesd:ciscat: INFO: Module disabled. Exiting...
2024/05/15 19:51:00 wazuh-modulesd:osquery: INFO: Module disabled. Exiting...
2024/05/15 19:51:00 sca: INFO: Module started.
2024/05/15 19:51:00 wazuh-modulesd:task-manager: INFO: (8200): Module Task Manager started.
2024/05/15 19:51:00 sca: INFO: Loaded policy '/var/ossec/ruleset/sca/cis_ubuntu22-04.yml'
2024/05/15 19:51:00 wazuh-modulesd:database: INFO: Module started.
2024/05/15 19:51:00 wazuh-modulesd:download: INFO: Module started.
2024/05/15 19:51:00 sca: INFO: Starting Security Configuration Assessment scan.
2024/05/15 19:51:00 wazuh-modulesd:control: INFO: Starting control thread.
2024/05/15 19:51:00 wazuh-modulesd: ERROR: (1401): Error reading authentication key: '
'.
2024/05/15 19:51:00 wazuh-modulesd:syscollector: INFO: Module started.
2024/05/15 19:51:00 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2024/05/15 19:51:00 sca: INFO: Starting evaluation of policy: '/var/ossec/ruleset/sca/cis_ubuntu22-04.yml'
2024/05/15 19:51:00 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2024/05/15 19:51:04 sca: INFO: Evaluation finished for policy '/var/ossec/ruleset/sca/cis_ubuntu22-04.yml'
2024/05/15 19:51:04 sca: INFO: Security Configuration Assessment scan finished. Duration: 4 seconds.
2024/05/15 19:51:17 rootcheck: INFO: Ending rootcheck scan.
2024/05/15 20:05:35 wazuh-authd: INFO: New connection from 172.17.0.2
2024/05/15 20:05:35 wazuh-authd: INFO: Received request for a new agent (ebbee3838a64) from: 172.17.0.2
2024/05/15 20:05:35 wazuh-authd: INFO: Agent key generated for 'ebbee3838a64' (requested by any)
2024/05/15 20:05:35 wazuh-authd: INFO: New connection from 172.17.0.3
2024/05/15 20:05:35 wazuh-authd: INFO: Received request for a new agent (0d135f67336f) from: 172.17.0.3
2024/05/15 20:05:35 wazuh-authd: INFO: Agent key generated for '0d135f67336f' (requested by any)
2024/05/15 20:05:37 wazuh-authd: INFO: New connection from 172.17.0.4
2024/05/15 20:05:37 wazuh-authd: INFO: Received request for a new agent (5af2147bf851) from: 172.17.0.4
2024/05/15 20:05:37 wazuh-authd: INFO: Agent key generated for '5af2147bf851' (requested by any)
2024/05/15 20:05:37 wazuh-authd: INFO: New connection from 172.17.0.5
2024/05/15 20:05:37 wazuh-authd: INFO: Received request for a new agent (655badc0a66e) from: 172.17.0.5
2024/05/15 20:05:37 wazuh-authd: INFO: Agent key generated for '655badc0a66e' (requested by any)
2024/05/15 20:05:37 wazuh-authd: INFO: New connection from 172.17.0.6
2024/05/15 20:05:37 wazuh-authd: INFO: Received request for a new agent (5b7a93e14cf7) from: 172.17.0.6
2024/05/15 20:05:37 wazuh-authd: INFO: Agent key generated for '5b7a93e14cf7' (requested by any)
2024/05/15 20:05:37 wazuh-authd: INFO: New connection from 172.17.0.10
2024/05/15 20:05:37 wazuh-authd: INFO: Received request for a new agent (05c8b8df823b) from: 172.17.0.10
2024/05/15 20:05:37 wazuh-authd: INFO: Agent key generated for '05c8b8df823b' (requested by any)
2024/05/15 20:05:37 wazuh-authd: INFO: New connection from 172.17.0.7
2024/05/15 20:05:37 wazuh-authd: INFO: Received request for a new agent (31a4e0679bde) from: 172.17.0.7
2024/05/15 20:05:37 wazuh-authd: INFO: Agent key generated for '31a4e0679bde' (requested by any)
2024/05/15 20:05:38 wazuh-authd: INFO: New connection from 172.17.0.8
2024/05/15 20:05:38 wazuh-authd: INFO: Received request for a new agent (b7334901b23f) from: 172.17.0.8
2024/05/15 20:05:38 wazuh-authd: INFO: Agent key generated for 'b7334901b23f' (requested by any)
2024/05/15 20:05:38 wazuh-authd: INFO: New connection from 172.17.0.9
2024/05/15 20:05:38 wazuh-authd: INFO: Received request for a new agent (97e71fae8dfa) from: 172.17.0.9
2024/05/15 20:05:38 wazuh-authd: INFO: Agent key generated for '97e71fae8dfa' (requested by any)
2024/05/15 20:05:38 wazuh-remoted: INFO: (1409): Authentication file changed. Updating.
2024/05/15 20:05:38 wazuh-remoted: INFO: (1410): Reading authentication keys file.
2024/05/15 20:05:38 wazuh-authd: INFO: New connection from 172.17.0.14
2024/05/15 20:05:38 wazuh-authd: INFO: Received request for a new agent (f58fd050166f) from: 172.17.0.14
2024/05/15 20:05:38 wazuh-authd: INFO: Agent key generated for 'f58fd050166f' (requested by any)
2024/05/15 20:05:38 wazuh-authd: INFO: New connection from 172.17.0.11
2024/05/15 20:05:38 wazuh-authd: INFO: Received request for a new agent (5280fd1c4348) from: 172.17.0.11
2024/05/15 20:05:38 wazuh-authd: INFO: Agent key generated for '5280fd1c4348' (requested by any)
2024/05/15 20:05:39 wazuh-authd: INFO: New connection from 172.17.0.15
2024/05/15 20:05:39 wazuh-authd: INFO: Received request for a new agent (5ecc189ba82d) from: 172.17.0.15
2024/05/15 20:05:39 wazuh-authd: INFO: Agent key generated for '5ecc189ba82d' (requested by any)
2024/05/15 20:05:39 wazuh-authd: INFO: New connection from 172.17.0.12
2024/05/15 20:05:39 wazuh-authd: INFO: Received request for a new agent (d31bd92a785a) from: 172.17.0.12
2024/05/15 20:05:39 wazuh-authd: INFO: Agent key generated for 'd31bd92a785a' (requested by any)
2024/05/15 20:05:39 wazuh-authd: INFO: New connection from 172.17.0.13
2024/05/15 20:05:39 wazuh-authd: INFO: Received request for a new agent (2b2509598d48) from: 172.17.0.13
2024/05/15 20:05:39 wazuh-authd: INFO: Agent key generated for '2b2509598d48' (requested by any)
2024/05/15 20:05:39 wazuh-authd: INFO: New connection from 172.17.0.17
2024/05/15 20:05:39 wazuh-authd: INFO: Received request for a new agent (b4a7e77d1767) from: 172.17.0.17
2024/05/15 20:05:39 wazuh-authd: INFO: Agent key generated for 'b4a7e77d1767' (requested by any)
2024/05/15 20:05:40 wazuh-authd: INFO: New connection from 172.17.0.18
2024/05/15 20:05:40 wazuh-authd: INFO: Received request for a new agent (2c2bfc3a2729) from: 172.17.0.18
2024/05/15 20:05:40 wazuh-authd: INFO: Agent key generated for '2c2bfc3a2729' (requested by any)
2024/05/15 20:05:40 wazuh-authd: INFO: New connection from 172.17.0.16
2024/05/15 20:05:40 wazuh-authd: INFO: Received request for a new agent (0b1e121a7885) from: 172.17.0.16
2024/05/15 20:05:40 wazuh-authd: INFO: Agent key generated for '0b1e121a7885' (requested by any)
2024/05/15 20:05:41 wazuh-authd: INFO: New connection from 172.17.0.19
2024/05/15 20:05:41 wazuh-authd: INFO: Received request for a new agent (4cb7e63edb8c) from: 172.17.0.19
2024/05/15 20:05:41 wazuh-authd: INFO: Agent key generated for '4cb7e63edb8c' (requested by any)
2024/05/15 20:05:41 wazuh-authd: INFO: New connection from 172.17.0.23
2024/05/15 20:05:41 wazuh-authd: INFO: Received request for a new agent (fb22990f463f) from: 172.17.0.23
2024/05/15 20:05:41 wazuh-authd: INFO: Agent key generated for 'fb22990f463f' (requested by any)
2024/05/15 20:05:41 wazuh-authd: INFO: New connection from 172.17.0.20
2024/05/15 20:05:41 wazuh-authd: INFO: Received request for a new agent (fa757f9d6952) from: 172.17.0.20
2024/05/15 20:05:41 wazuh-authd: INFO: Agent key generated for 'fa757f9d6952' (requested by any)
2024/05/15 20:05:41 wazuh-authd: INFO: New connection from 172.17.0.21
2024/05/15 20:05:41 wazuh-authd: INFO: Received request for a new agent (0ae31201f220) from: 172.17.0.21
2024/05/15 20:05:41 wazuh-authd: INFO: Agent key generated for '0ae31201f220' (requested by any)
2024/05/15 20:05:42 wazuh-authd: INFO: New connection from 172.17.0.22
2024/05/15 20:05:42 wazuh-authd: INFO: Received request for a new agent (54232eeafc87) from: 172.17.0.22
2024/05/15 20:05:42 wazuh-authd: INFO: Agent key generated for '54232eeafc87' (requested by any)
2024/05/15 20:05:42 wazuh-authd: INFO: New connection from 172.17.0.27
2024/05/15 20:05:42 wazuh-authd: INFO: Received request for a new agent (033d43363cf9) from: 172.17.0.27
2024/05/15 20:05:42 wazuh-authd: INFO: Agent key generated for '033d43363cf9' (requested by any)
2024/05/15 20:05:42 wazuh-authd: INFO: New connection from 172.17.0.24
2024/05/15 20:05:42 wazuh-authd: INFO: Received request for a new agent (0ac15bbd08e4) from: 172.17.0.24
2024/05/15 20:05:42 wazuh-authd: INFO: Agent key generated for '0ac15bbd08e4' (requested by any)
2024/05/15 20:05:42 wazuh-authd: INFO: New connection from 172.17.0.29
2024/05/15 20:05:42 wazuh-authd: INFO: Received request for a new agent (4f4e008c87d8) from: 172.17.0.29
2024/05/15 20:05:42 wazuh-authd: INFO: Agent key generated for '4f4e008c87d8' (requested by any)
2024/05/15 20:05:42 wazuh-authd: INFO: New connection from 172.17.0.25
2024/05/15 20:05:42 wazuh-authd: INFO: Received request for a new agent (2a341240836b) from: 172.17.0.25
2024/05/15 20:05:42 wazuh-authd: INFO: Agent key generated for '2a341240836b' (requested by any)
2024/05/15 20:05:43 wazuh-authd: INFO: New connection from 172.17.0.26
2024/05/15 20:05:43 wazuh-authd: INFO: Received request for a new agent (e895711856df) from: 172.17.0.26
2024/05/15 20:05:43 wazuh-authd: INFO: Agent key generated for 'e895711856df' (requested by any)
2024/05/15 20:05:43 wazuh-authd: INFO: New connection from 172.17.0.31
2024/05/15 20:05:43 wazuh-authd: INFO: Received request for a new agent (aa15e24bfff8) from: 172.17.0.31
2024/05/15 20:05:43 wazuh-authd: INFO: Agent key generated for 'aa15e24bfff8' (requested by any)
2024/05/15 20:05:43 wazuh-authd: INFO: New connection from 172.17.0.32
2024/05/15 20:05:43 wazuh-authd: INFO: Received request for a new agent (4070d8dd74fa) from: 172.17.0.32
2024/05/15 20:05:43 wazuh-authd: INFO: Agent key generated for '4070d8dd74fa' (requested by any)
2024/05/15 20:05:43 wazuh-authd: INFO: New connection from 172.17.0.28
2024/05/15 20:05:43 wazuh-authd: INFO: Received request for a new agent (6b487cb982b7) from: 172.17.0.28
2024/05/15 20:05:43 wazuh-authd: INFO: Agent key generated for '6b487cb982b7' (requested by any)
2024/05/15 20:05:43 wazuh-authd: INFO: New connection from 172.17.0.34
2024/05/15 20:05:43 wazuh-authd: INFO: Received request for a new agent (cd8e98cc48ff) from: 172.17.0.34
2024/05/15 20:05:43 wazuh-authd: INFO: Agent key generated for 'cd8e98cc48ff' (requested by any)
2024/05/15 20:05:44 wazuh-authd: INFO: New connection from 172.17.0.30
2024/05/15 20:05:44 wazuh-authd: INFO: Received request for a new agent (97b0a0f890ed) from: 172.17.0.30
2024/05/15 20:05:44 wazuh-authd: INFO: Agent key generated for '97b0a0f890ed' (requested by any)
2024/05/15 20:05:44 wazuh-authd: INFO: New connection from 172.17.0.35
2024/05/15 20:05:44 wazuh-authd: INFO: Received request for a new agent (cb989ffbdb4e) from: 172.17.0.35
2024/05/15 20:05:44 wazuh-authd: INFO: Agent key generated for 'cb989ffbdb4e' (requested by any)
2024/05/15 20:05:44 wazuh-authd: INFO: New connection from 172.17.0.33
2024/05/15 20:05:44 wazuh-authd: INFO: Received request for a new agent (aaf2b39e884d) from: 172.17.0.33
2024/05/15 20:05:44 wazuh-authd: INFO: Agent key generated for 'aaf2b39e884d' (requested by any)
2024/05/15 20:05:44 wazuh-authd: INFO: New connection from 172.17.0.38
2024/05/15 20:05:44 wazuh-authd: INFO: Received request for a new agent (abf8c8325d0c) from: 172.17.0.38
2024/05/15 20:05:44 wazuh-authd: INFO: Agent key generated for 'abf8c8325d0c' (requested by any)
2024/05/15 20:05:45 wazuh-authd: INFO: New connection from 172.17.0.39
2024/05/15 20:05:45 wazuh-authd: INFO: Received request for a new agent (0ab3456c7750) from: 172.17.0.39
2024/05/15 20:05:45 wazuh-authd: INFO: Agent key generated for '0ab3456c7750' (requested by any)
2024/05/15 20:05:45 wazuh-authd: INFO: New connection from 172.17.0.40
2024/05/15 20:05:45 wazuh-authd: INFO: Received request for a new agent (9544cfb6653f) from: 172.17.0.40
2024/05/15 20:05:45 wazuh-authd: INFO: Agent key generated for '9544cfb6653f' (requested by any)
2024/05/15 20:05:45 wazuh-authd: INFO: New connection from 172.17.0.36
2024/05/15 20:05:45 wazuh-authd: INFO: Received request for a new agent (462a86c4cc01) from: 172.17.0.36
2024/05/15 20:05:45 wazuh-authd: INFO: Agent key generated for '462a86c4cc01' (requested by any)
2024/05/15 20:05:45 wazuh-authd: INFO: New connection from 172.17.0.37
2024/05/15 20:05:45 wazuh-authd: INFO: Received request for a new agent (0633dfe21a1d) from: 172.17.0.37
2024/05/15 20:05:45 wazuh-authd: INFO: Agent key generated for '0633dfe21a1d' (requested by any)
2024/05/15 20:05:45 wazuh-authd: INFO: New connection from 172.17.0.42
2024/05/15 20:05:45 wazuh-authd: INFO: Received request for a new agent (4ae66790d7b6) from: 172.17.0.42
2024/05/15 20:05:45 wazuh-authd: INFO: Agent key generated for '4ae66790d7b6' (requested by any)
2024/05/15 20:05:45 wazuh-authd: INFO: New connection from 172.17.0.43
2024/05/15 20:05:45 wazuh-authd: INFO: Received request for a new agent (b14d4f05cd6d) from: 172.17.0.43
2024/05/15 20:05:45 wazuh-authd: INFO: Agent key generated for 'b14d4f05cd6d' (requested by any)
2024/05/15 20:05:46 wazuh-authd: INFO: New connection from 172.17.0.44
2024/05/15 20:05:46 wazuh-authd: INFO: Received request for a new agent (6b2f63ee51eb) from: 172.17.0.44
2024/05/15 20:05:46 wazuh-authd: INFO: Agent key generated for '6b2f63ee51eb' (requested by any)
2024/05/15 20:05:46 wazuh-authd: INFO: New connection from 172.17.0.41
2024/05/15 20:05:46 wazuh-authd: INFO: Received request for a new agent (fa53ac5d0b15) from: 172.17.0.41
2024/05/15 20:05:46 wazuh-authd: INFO: Agent key generated for 'fa53ac5d0b15' (requested by any)
2024/05/15 20:05:46 wazuh-authd: INFO: New connection from 172.17.0.47
2024/05/15 20:05:46 wazuh-authd: INFO: Received request for a new agent (1c8733ae4a72) from: 172.17.0.47
2024/05/15 20:05:46 wazuh-authd: INFO: Agent key generated for '1c8733ae4a72' (requested by any)
2024/05/15 20:05:47 wazuh-authd: INFO: New connection from 172.17.0.48
2024/05/15 20:05:47 wazuh-authd: INFO: Received request for a new agent (8ef5ead90721) from: 172.17.0.48
2024/05/15 20:05:47 wazuh-authd: INFO: Agent key generated for '8ef5ead90721' (requested by any)
2024/05/15 20:05:47 wazuh-authd: INFO: New connection from 172.17.0.45
2024/05/15 20:05:47 wazuh-authd: INFO: Received request for a new agent (592a28ef02a9) from: 172.17.0.45
2024/05/15 20:05:47 wazuh-authd: INFO: Agent key generated for '592a28ef02a9' (requested by any)
2024/05/15 20:05:47 wazuh-authd: INFO: New connection from 172.17.0.46
2024/05/15 20:05:47 wazuh-authd: INFO: Received request for a new agent (bff89f651551) from: 172.17.0.46
2024/05/15 20:05:47 wazuh-authd: INFO: Agent key generated for 'bff89f651551' (requested by any)
2024/05/15 20:05:48 wazuh-authd: INFO: New connection from 172.17.0.49
2024/05/15 20:05:48 wazuh-authd: INFO: Received request for a new agent (fd7499cf8e80) from: 172.17.0.49
2024/05/15 20:05:48 wazuh-authd: INFO: Agent key generated for 'fd7499cf8e80' (requested by any)
2024/05/15 20:05:48 wazuh-remoted: INFO: (1409): Authentication file changed. Updating.
2024/05/15 20:05:48 wazuh-remoted: INFO: (1410): Reading authentication keys file.
2024/05/15 20:05:48 wazuh-authd: INFO: New connection from 172.17.0.50
2024/05/15 20:05:48 wazuh-authd: INFO: Received request for a new agent (b77d74f315e7) from: 172.17.0.50
2024/05/15 20:05:48 wazuh-authd: INFO: Agent key generated for 'b77d74f315e7' (requested by any)
2024/05/15 20:05:48 wazuh-authd: INFO: New connection from 172.17.0.51
2024/05/15 20:05:48 wazuh-authd: INFO: Received request for a new agent (2327e1ed5ad8) from: 172.17.0.51
2024/05/15 20:05:48 wazuh-authd: INFO: Agent key generated for '2327e1ed5ad8' (requested by any)
2024/05/15 20:05:58 wazuh-remoted: INFO: (1409): Authentication file changed. Updating.
2024/05/15 20:05:58 wazuh-remoted: INFO: (1410): Reading authentication keys file.
2024/05/15 20:05:59 wazuh-db: INFO: (1225): SIGNAL [(15)-(Terminated)] Received. Exit Cleaning...
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot receive message: Connection reset by peer (104)
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (104) 'Connection reset by peer'.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: No response from wazuh-db.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Error querying Wazuh DB to get the agent's 14 labels.
2024/05/15 20:06:00 wazuh-remoted: ERROR: Cannot receive message: Connection reset by peer (104)
2024/05/15 20:06:00 wazuh-remoted: ERROR: Cannot send message: (104) 'Connection reset by peer'.
2024/05/15 20:06:00 wazuh-remoted: WARNING: Unable to save last keepalive and set connection status as pending for agent: 002
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot receive message: Connection reset by peer (104)
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot receive message: Connection reset by peer (104)
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot receive message: Connection reset by peer (104)
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (104) 'Connection reset by peer'.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:00 wazuh-remoted: ERROR: Cannot receive message: Connection reset by peer (104)
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:00 wazuh-remoted: ERROR: Cannot send message: (104) 'Connection reset by peer'.
2024/05/15 20:06:00 wazuh-remoted: ERROR: No response from wazuh-db.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (104) 'Connection reset by peer'.
2024/05/15 20:06:00 wazuh-remoted: ERROR: Error querying Wazuh DB to get the agent's 2 group.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: FIM decoder: Cannot get response from database.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot receive message: Connection reset by peer (104)
2024/05/15 20:06:00 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot receive message: Connection reset by peer (104)
2024/05/15 20:06:00 wazuh-analysisd: ERROR: FIM decoder: Cannot get response from database.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (104) 'Connection reset by peer'.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: dbsync: Cannot get response from database.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:00 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot receive message: Connection reset by peer (104)
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:00 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:00 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (104) 'Connection reset by peer'.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: dbsync: Cannot get response from database.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (104) 'Connection reset by peer'.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot receive message: Connection reset by peer (104)
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (104) 'Connection reset by peer'.
2024/05/15 20:06:00 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:00 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:00 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: dbsync: Cannot get response from database.
2024/05/15 20:06:00 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:00 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Rootcheck decoder unexpected result: ''
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (104) 'Connection reset by peer'.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: dbsync: Cannot get response from database.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:00 wazuh-remoted: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:00 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:00 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:00 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:00 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:00 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:00 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:00 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:00 wazuh-modulesd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:01 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:01 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:01 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:01 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:01 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:01 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:01 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:01 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:01 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:01 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:01 wazuh-remoted: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:01 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:01 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:01 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:01 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:01 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:01 wazuh-modulesd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:02 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:02 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:02 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:03 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:03 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:03 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:03 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:03 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:03 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:03 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:03 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:03 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:03 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:03 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:03 wazuh-remoted: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:03 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:03 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:03 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:03 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:03 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:03 wazuh-modulesd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:04 wazuh-remoted: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:04 wazuh-remoted: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:04 wazuh-remoted: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:05 wazuh-remoted: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:05 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Error querying Wazuh DB to get the agent's 15 labels.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:06 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: dbsync: Cannot communicate with database.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: dbsync: Cannot communicate with database.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: dbsync: Cannot communicate with database.
2024/05/15 20:06:06 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: dbsync: Cannot communicate with database.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: dbsync: Cannot communicate with database.
2024/05/15 20:06:06 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:06 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:06 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:06 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: dbsync: Cannot communicate with database.
2024/05/15 20:06:06 wazuh-remoted: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-remoted: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: dbsync: Cannot communicate with database.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-remoted: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-remoted: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Bad load query: 'agent 012 rootcheck save 1715803560 Starting rootcheck scan.'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: FIM decoder: Cannot communicate with database.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Bad load query: 'agent 011 rootcheck save 1715803560 Starting rootcheck scan.'.
2024/05/15 20:06:06 wazuh-remoted: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:06 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: dbsync: Cannot communicate with database.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Rootcheck decoder unexpected result: ''
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: FIM decoder: Cannot communicate with database.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Rootcheck decoder unexpected result: ''
2024/05/15 20:06:06 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: FIM decoder: Cannot communicate with database.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-analysisd: ERROR: FIM decoder: Cannot communicate with database.
2024/05/15 20:06:06 wazuh-modulesd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-modulesd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:06 wazuh-modulesd:task-manager: ERROR: (8209): Tasks DB Cannot execute SQL query: err database 'queue/tasks/tasks.db'
2024/05/15 20:06:07 wazuh-remoted: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:07 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:07 wazuh-remoted: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:07 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:07 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:07 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:07 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:07 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:07 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:07 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:07 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:07 wazuh-remoted: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:08 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:08 wazuh-remoted: INFO: Cannot find 'queue/db/wdb'. Waiting 2 seconds to reconnect.
2024/05/15 20:06:09 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:09 wazuh-remoted: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:09 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:09 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:09 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:09 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:09 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:09 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:09 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:09 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:10 wazuh-remoted: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:10 wazuh-remoted: INFO: Cannot find 'queue/db/wdb'. Waiting 3 seconds to reconnect.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-remoted: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-remoted: ERROR: Unable to connect to socket 'queue/db/wdb'
2024/05/15 20:06:12 wazuh-remoted: ERROR: Error querying Wazuh DB to get the agent's 14 group.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: dbsync: Cannot communicate with database.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: dbsync: Cannot communicate with database.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: dbsync: Cannot communicate with database.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: dbsync: Cannot communicate with database.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:12 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:12 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: dbsync: Cannot communicate with database.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: dbsync: Cannot communicate with database.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: dbsync: Cannot communicate with database.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: dbsync: Cannot communicate with database.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Unable to connect to socket 'queue/db/wdb'
2024/05/15 20:06:12 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Error querying Wazuh DB to get the agent's 2 labels.
2024/05/15 20:06:12 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:12 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:12 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:12 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:13 wazuh-analysisd: INFO: Cannot find 'queue/db/wdb'. Waiting 1 seconds to reconnect.
2024/05/15 20:06:13 wazuh-db: INFO: Started (pid: 159266).
2024/05/15 20:06:13 wazuh-remoted: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:13 wazuh-remoted: ERROR: Unable to connect to socket 'queue/db/wdb'.
2024/05/15 20:06:13 wazuh-remoted: ERROR: Error querying Wazuh DB to get agent's groups.
2024/05/15 20:06:13 wazuh-remoted: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-remoted: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-remoted: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-remoted: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-remoted: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-remoted: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:13 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:06:14 wazuh-analysisd: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:06:14 wazuh-analysisd: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:11:12 wazuh-monitord: ERROR: Cannot send message: (32) 'Broken pipe'.
2024/05/15 20:11:12 wazuh-monitord: ERROR: Connection with wazuh-db lost. Reconnecting.
2024/05/15 20:51:01 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2024/05/15 20:52:08 wazuh-modulesd:syscollector: INFO: Evaluation finished.
/var/ossec/queue/db/root@vm-ubuntu2204-server:/home/vagrant# ls -l /var/ossec/queue/db/
total 30372
-rw-r----- 1 wazuh wazuh 2650112 May 15 21:46 000.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 001.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 002.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 003.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 004.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 005.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 006.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 007.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 008.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 009.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 010.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 011.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 012.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 013.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 014.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 015.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 016.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 017.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 018.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 019.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 020.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 021.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 022.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 023.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 024.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 025.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 026.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 027.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 028.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 029.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 030.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 031.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 032.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 033.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 034.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 035.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 036.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 037.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 038.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 039.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 040.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 041.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 042.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 043.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 044.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 045.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 046.db
-rw-r----- 1 wazuh wazuh 557056 May 15 21:46 047.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 048.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 049.db
-rw-r----- 1 wazuh wazuh 569344 May 15 21:46 050.db
-rw-r----- 1 wazuh wazuh 131072 May 15 21:47 global.db
-rw-r----- 1 wazuh wazuh 66176 May 15 21:47 global.db-journal
srw-rw---- 1 wazuh wazuh 0 May 15 20:06 wdb |
Test on v4.7.3 🔴 :Replicated after following the steps. root@kinetic:/home/vagrant/workspace/wazuh/src# /var/ossec/bin/wazuh-control info
WAZUH_VERSION="v4.7.3"
WAZUH_REVISION="40714"
WAZUH_TYPE="server"Just by enrolling 10 agents the issue appears: root@kinetic:/home/vagrant/workspace/wazuh/src# ls -l /var/ossec/queue/db/
total 10200
-rw-r----- 1 wazuh wazuh 3461120 May 15 17:52 000.db
-rw-r----- 1 wazuh wazuh 144152 May 15 17:52 000.db-journal
-rw-r----- 1 wazuh wazuh 757760 May 15 17:49 001.db
-rw-r----- 1 wazuh wazuh 757760 May 15 17:49 002.db
-rw-r----- 1 wazuh wazuh 757760 May 15 17:49 003.db
-rw-r----- 1 wazuh wazuh 757760 May 15 17:49 004.db
-rw-r----- 1 wazuh wazuh 757760 May 15 17:49 005.db
-rw-r----- 1 wazuh wazuh 757760 May 15 17:49 006.db
-rw-r----- 1 wazuh wazuh 757760 May 15 17:49 007.db
-rw-r----- 1 wazuh wazuh 704512 May 15 17:49 008.db
-rw-r----- 1 wazuh wazuh 0 May 15 17:48 009.db
-rw-r----- 1 wazuh wazuh 704512 May 15 17:49 010.db
-rw-r----- 1 wazuh wazuh 90112 May 15 17:52 global.db
-rw-r----- 1 wazuh wazuh 21032 May 15 17:52 global.db-journal
srw-rw---- 1 wazuh wazuh 0 May 15 17:48 wdb
Thread Sanitizer report/var/ossec/bin/wazuh-db-modif -f2024/05/15 17:48:23 wazuh-db: INFO: Started (pid: 34447).
==================
WARNING: ThreadSanitizer: data race (pid=34447)
Write of size 8 at 0x7b7000010f68 by thread T10:
#0 wdb_pool_leave wazuh_db/wdb_pool.c:76 (wazuh-db-ts+0x599c4)
#1 wdb_commit_old wazuh_db/wdb.c:938 (wazuh-db-ts+0x19b98)
#2 run_gc wazuh_db/main.c:423 (wazuh-db-ts+0x1526e)
Previous write of size 8 at 0x7b7000010f68 by thread T11:
#0 wdb_pool_leave wazuh_db/wdb_pool.c:76 (wazuh-db-ts+0x599c4)
#1 run_up wazuh_db/main.c:520 (wazuh-db-ts+0x15896)
As if synchronized via sleep:
#0 sleep ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:344 (libtsan.so.2+0x678be)
#1 run_gc wazuh_db/main.c:434 (wazuh-db-ts+0x152b0)
Location is heap block of size 1976 at 0x7b7000010800 allocated by thread T11:
#0 calloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:667 (libtsan.so.2+0x3fde2)
#1 wdb_init wazuh_db/wdb.c:885 (wazuh-db-ts+0x19648)
#2 wdb_pool_get_or_create wazuh_db/wdb_pool.c:56 (wazuh-db-ts+0x596cb)
#3 wdb_open_agent2 wazuh_db/wdb.c:398 (wazuh-db-ts+0x17469)
#4 run_up wazuh_db/main.c:517 (wazuh-db-ts+0x1587f)
Thread T10 (tid=34458, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1001 (libtsan.so.2+0x63a69)
#1 main wazuh_db/main.c:218 (wazuh-db-ts+0x1427c)
Thread T11 (tid=34459, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1001 (libtsan.so.2+0x63a69)
#1 main wazuh_db/main.c:223 (wazuh-db-ts+0x142ed)
SUMMARY: ThreadSanitizer: data race wazuh_db/wdb_pool.c:76 in wdb_pool_leave
==================
==================
WARNING: ThreadSanitizer: data race (pid=34447)
Read of size 8 at 0x7b7000020768 by thread T10 (mutexes: write M21):
#0 wdb_commit_old wazuh_db/wdb.c:928 (wazuh-db-ts+0x19a7a)
#1 run_gc wazuh_db/main.c:423 (wazuh-db-ts+0x1526e)
Previous write of size 8 at 0x7b7000020768 by thread T2:
#0 wdb_pool_leave wazuh_db/wdb_pool.c:76 (wazuh-db-ts+0x599c4)
#1 wdb_parse wazuh_db/wdb_parser.c:1405 (wazuh-db-ts+0x3f438)
#2 run_worker wazuh_db/main.c:397 (wazuh-db-ts+0x15049)
As if synchronized via sleep:
#0 sleep ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:344 (libtsan.so.2+0x678be)
#1 run_gc wazuh_db/main.c:434 (wazuh-db-ts+0x152b0)
Location is heap block of size 1976 at 0x7b7000020000 allocated by thread T2:
#0 calloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:667 (libtsan.so.2+0x3fde2)
#1 wdb_init wazuh_db/wdb.c:885 (wazuh-db-ts+0x19648)
#2 wdb_pool_get_or_create wazuh_db/wdb_pool.c:56 (wazuh-db-ts+0x596cb)
#3 wdb_open_global wazuh_db/wdb.c:333 (wazuh-db-ts+0x17018)
#4 wdb_parse wazuh_db/wdb_parser.c:871 (wazuh-db-ts+0x39514)
#5 run_worker wazuh_db/main.c:397 (wazuh-db-ts+0x15049)
Mutex M21 (0x7b7000020778) created at:
#0 pthread_mutex_init ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1295 (libtsan.so.2+0x53cb6)
#1 wdb_init wazuh_db/wdb.c:886 (wazuh-db-ts+0x196cb)
#2 wdb_pool_get_or_create wazuh_db/wdb_pool.c:56 (wazuh-db-ts+0x596cb)
#3 wdb_open_global wazuh_db/wdb.c:333 (wazuh-db-ts+0x17018)
#4 wdb_parse wazuh_db/wdb_parser.c:871 (wazuh-db-ts+0x39514)
#5 run_worker wazuh_db/main.c:397 (wazuh-db-ts+0x15049)
Thread T10 (tid=34458, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1001 (libtsan.so.2+0x63a69)
#1 main wazuh_db/main.c:218 (wazuh-db-ts+0x1427c)
Thread T2 (tid=34450, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1001 (libtsan.so.2+0x63a69)
#1 main wazuh_db/main.c:212 (wazuh-db-ts+0x141d7)
SUMMARY: ThreadSanitizer: data race wazuh_db/wdb.c:928 in wdb_commit_old
==================
==================
WARNING: ThreadSanitizer: data race (pid=34447)
Write of size 8 at 0x7b7000030768 by thread T3:
#0 wdb_pool_leave wazuh_db/wdb_pool.c:76 (wazuh-db-ts+0x599c4)
#1 wdb_parse wazuh_db/wdb_parser.c:771 (wazuh-db-ts+0x388c6)
#2 run_worker wazuh_db/main.c:397 (wazuh-db-ts+0x15049)
Previous write of size 8 at 0x7b7000030768 by thread T9:
#0 wdb_pool_leave wazuh_db/wdb_pool.c:76 (wazuh-db-ts+0x599c4)
#1 wdb_parse wazuh_db/wdb_parser.c:771 (wazuh-db-ts+0x388c6)
#2 run_worker wazuh_db/main.c:397 (wazuh-db-ts+0x15049)
Location is heap block of size 1976 at 0x7b7000030000 allocated by thread T8:
#0 calloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:667 (libtsan.so.2+0x3fde2)
#1 wdb_init wazuh_db/wdb.c:885 (wazuh-db-ts+0x19648)
#2 wdb_pool_get_or_create wazuh_db/wdb_pool.c:56 (wazuh-db-ts+0x596cb)
#3 wdb_open_agent2 wazuh_db/wdb.c:398 (wazuh-db-ts+0x17469)
#4 wdb_parse wazuh_db/wdb_parser.c:288 (wazuh-db-ts+0x33503)
#5 run_worker wazuh_db/main.c:397 (wazuh-db-ts+0x15049)
Thread T3 (tid=34451, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1001 (libtsan.so.2+0x63a69)
#1 main wazuh_db/main.c:212 (wazuh-db-ts+0x141d7)
Thread T9 (tid=34457, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1001 (libtsan.so.2+0x63a69)
#1 main wazuh_db/main.c:212 (wazuh-db-ts+0x141d7)
Thread T8 (tid=34456, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1001 (libtsan.so.2+0x63a69)
#1 main wazuh_db/main.c:212 (wazuh-db-ts+0x141d7)
SUMMARY: ThreadSanitizer: data race wazuh_db/wdb_pool.c:76 in wdb_pool_leave
Logsossec.log2024/05/15 17:48:25 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(88): no such table: sync_info
2024/05/15 17:48:25 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(211): no such table: sys_hwinfo
2024/05/15 17:48:25 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(71): no such table: sca_policy
2024/05/15 17:48:25 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(71): no such table: sca_policy
2024/05/15 17:48:25 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(88): no such table: sync_info
2024/05/15 17:48:25 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(211): no such table: sys_hwinfo
2024/05/15 17:48:25 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(88): no such table: sync_info
2024/05/15 17:48:25 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(217): no such table: sys_osinfo
2024/05/15 17:48:25 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(88): no such table: sync_info
2024/05/15 17:48:25 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(205): no such table: sys_netiface
2024/05/15 17:48:25 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(88): no such table: sync_info
2024/05/15 17:48:25 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(193): no such table: sys_netproto
2024/05/15 17:48:25 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(88): no such table: sync_info
2024/05/15 17:48:25 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(199): no such table: sys_netaddr
2024/05/15 17:48:25 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(88): no such table: sync_info
2024/05/15 17:48:25 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(175): no such table: sys_programs
2024/05/15 17:48:26 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(184): no such table: sys_hotfixes
2024/05/15 17:48:26 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(191): no such table: sys_ports
2024/05/15 17:48:26 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(88): no such table: sync_info
2024/05/15 17:48:26 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(169): no such table: sys_processes
2024/05/15 17:48:29 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(112): no such table: pm_event
2024/05/15 17:48:29 wazuh-db: ERROR: DB(009) Cannot cache statement
2024/05/15 17:48:29 wazuh-db: ERROR: DB(009) Error updating rootcheck PM tuple on SQLite database
2024/05/15 17:48:29 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(46): no such table: scan_info
2024/05/15 17:48:29 wazuh-db: ERROR: DB(009) Cannot cache statement
2024/05/15 17:48:29 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(47): no such table: scan_info
2024/05/15 17:48:29 wazuh-db: ERROR: DB(009) Cannot cache statement
2024/05/15 17:48:29 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(88): no such table: sync_info
2024/05/15 17:48:29 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(90): no such table: fim_entry
2024/05/15 17:48:38 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(112): no such table: pm_event
2024/05/15 17:48:38 wazuh-db: ERROR: DB(009) Cannot cache statement
2024/05/15 17:48:38 wazuh-db: ERROR: DB(009) Error updating rootcheck PM tuple on SQLite database
2024/05/15 17:53:29 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(88): no such table: sync_info
2024/05/15 17:53:29 wazuh-db: ERROR: DB(009) sqlite3_prepare_v2() stmt(90): no such table: fim_entry
|
Test on v4.8.0 🔴 :Replicated after following the steps. /var/ossec/bin/wazuh-control info
WAZUH_VERSION="v4.8.0"
WAZUH_REVISION="40809"
WAZUH_TYPE="server"Enrolling 50 agents the issue appears: Thread Sanitizer failsls -l /var/ossec/bin/wazuh-db*
-rwxr-x--- 1 root root 3979448 may 15 18:46 /var/ossec/bin/wazuh-db
-rwxr-x--- 1 root root 3129800 may 15 18:46 /var/ossec/bin/wazuh-dbd
-rwxr-xr-x 1 root root 4094504 may 15 19:16 /var/ossec/bin/wazuh-db-ts
/var/ossec/bin/wazuh-db-ts -f
FATAL: ThreadSanitizer: unexpected memory mapping 0x59a67c207000-0x59a67c216000Logsossec.log2024/05/15 17:50:11 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(111): no such table: pm_event
2024/05/15 17:50:11 wazuh-db: ERROR: DB(002) Cannot cache statement
2024/05/15 17:50:11 wazuh-db: ERROR: DB(002) Error updating rootcheck PM tuple on SQLite database
2024/05/15 17:50:14 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(70): no such table: sca_policy
2024/05/15 17:50:14 wazuh-analysisd: ERROR: Error querying policy monitoring database for agent '002'
2024/05/15 17:50:16 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/15 17:50:16 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(211): no such table: sys_hwinfo
2024/05/15 17:50:16 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/15 17:50:16 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/15 17:50:16 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(217): no such table: sys_osinfo
2024/05/15 17:50:16 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/15 17:50:16 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/15 17:50:16 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(205): no such table: sys_netiface
2024/05/15 17:50:16 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/15 17:50:16 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_infoTablesNormal table sqlite3 /var/ossec/queue/db/032.db
sqlite> .tables
ciscat_results sca_policy sys_netiface
fim_entry sca_scan_info sys_netproto
metadata scan_info sys_osinfo
pm_event sync_info sys_ports
sca_check sys_hotfixes sys_processes
sca_check_compliance sys_hwinfo sys_programs
sca_check_rules sys_netaddr
Corrupted table sqlite3 /var/ossec/queue/db/002.db
sqlite> .tables
fim_entry
|
|
Based on the results of the tests we performed with the team, we obtained the following answers. The origin of this bug
The impact
Probability of occurrence
Estimation
|
ThreadSanitizer report 1Data race 1Data race 2Data race 3Data race 4Data race 5Data race 6ThreadSanitizer report 2Data race 7 |
Bug cause addressedRationaleWhen wazuh-db starts, it deletes the template to handle upgrades. As new agents connect, the template is copied. If the template doesn't exist, it is created. This process is handled by 8 threads simultaneously, causing each thread to copy half of the new template that the following thread is creating. This explains that:
None of the defects found by ThreadSanitizer is related to this bug. In fact, I'm pretty sure they have no impact. Fix proposal
|
5 tasks
5 tasks
vikman90
added
the
type/bug/regression
This was referenced May 17, 2024
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
During the execution of the automated Vulnerability Detector tests at #22826, it was found that the vulnerabilities were not generated correctly. Checking the environment logs we have been able to see:
The worker log only appears once, but the master logs appear continuously and never stabilize.
The text was updated successfully, but these errors were encountered: