Release 4.8.0 - RC 1 - Vulnerability Detection E2E tests

[juliamagan]

Tests information

Main release stage issue	#23246
Main Vulnerability Detection tests issue	#23298
Version	4.8.0
Release stage	RC 1
Tag	https://github.com/wazuh/wazuh-qa/tree/v4.8.0-rc1
Previous E2E Vulnerability tests issue	#23060

Description

The following issue aims to run all Vulnerability Detector E2E tests for the current release candidate, report the results, and open new issues for any encountered errors.

Instruction

Run the vulnerability detector tests using the Test_e2e_system pipeline:

Conclusion 🔴

Report: https://github.com/wazuh/wazuh/files/15234960/Test_e2e_system_284_test_vulnerability_detector.zip

• Broken database during Vulnerability Detector tests #22847
• Vulnerability Detector fails to identify ARM centOS vulnerabilities in End-to-End tests #23326
• Inconsistent results detected by vulnerability Detector in E2E Tests #23327
• Vulnerability Detector does not detect vulnerabilities in certain agents in E2E tests #23328
• Failure installing upgrade_package_nonvulnerable_to_vulnerable windows agent test case in E2E VD tests wazuh-qa#5344
• Agents are not being restarted after manager change in E2E vulnerability detection tests wazuh-qa#5345

[Rebits]

After talking with @juliamagan we have decided to launch these tests twice, with the v4.8.0-rc1 tag branch and with recent fixes reported in wazuh/wazuh-qa#5319

• v4.8.0-rc1: https://ci.wazuh.info/job/Test_e2e_system/280
• vd-global-fixes: https://ci.wazuh.info/job/Test_e2e_system/284

---

E2E tests fail for v4.8.0-rc1 due to https://github.com/wazuh/wazuh-jenkins/issues/6445. This was fixed in vd-global-fixes branch

[Rebits]

Vulnerability Detection E2E Tests 🔴

Report: Test_e2e_system_284_test_vulnerability_detector.zip

Failed cases

All test cases fail:

• test_first_syscollector_scan[vd_disabled_when_agents_registration
• test_first_syscollector_scan[vd_enabled_when_agents_registration]
• test_consistency_initial_scans
• test_syscollector_second_scan
• test_install_vulnerable_package_when_agent_down[install_package]
• test_change_agent_manager[install_package]
• test_vulnerability_detector_scans_cases (All cases)

General Failures 🔴

• Broken database. The proposed fix does not solve the issue Create agent database files from template atomically #23216. The issue was reopened Broken database during Vulnerability Detector tests #22847

2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(211): no such table: sys_hwinfo
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(217): no such table: sys_osinfo
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(205): no such table: sys_netiface
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(193): no such table: sys_netproto
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(199): no such table: sys_netaddr
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(175): no such table: sys_programs
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(184): no such table: sys_hotfixes
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(187): no such table: sys_ports
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(169): no such table: sys_processes
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_hwinfo
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_osinfo
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_netiface
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:32 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
2024/05/07 09:12:32 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:32 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
2024/05/07 09:12:32 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:32 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
...

Initial Scan Tests 🔴

• Vulnerability detection does not detect any vulnerability for the centOS agent (agent4) at the initial scans: Vulnerability Detector fails to identify ARM centOS vulnerabilities in End-to-End tests #23326
• Vulnerabilities detected before and after the vulnerability detection module is enabled are not consistent: Inconsistent results detected by vulnerability Detector in E2E Tests #23327

Install/Upgrade/Removal cases 🔴

• No vulnerabilities were triggered in agent agent4 for test_install_vulnerable_package_when_agent_down case. Reported in Vulnerability Detector does not detect vulnerabilities in certain agents in E2E tests #23328
• No vulnerabilities or alerts were triggered in agents agent3, agent5, agent2 in all cases. Reported in Vulnerability Detector does not detect vulnerabilities in certain agents in E2E tests #23328

Evidence collection 🔴

The test case test_change_agent_manager[install_package] truncates logs after the agent's manager change, providing very little evidence to troubleshoot failures in this test. Reported in wazuh/wazuh-qa#5345

Environment unstable 🔴

In the upgrade_package_nonvulnerable_to_vulnerable test case, it has failed the upgrade of the vulnerable package for Windows agent:

2024-05-07 11:11:32 Error installing the package on agent2: Failed to install the package in agent2: {'changed': False, 'msg': 'Unhandled exception while executing module: Exception calling "OpenPackage" with "2" argument(s): "This installation package could not be opened. Verify that the package exists and that you can access it, or contact the application vendor to verify that this is a valid Windows Installer package"'} (remote_operations_handler.py:401)

Reported in wazuh/wazuh-qa#5344

[MARCOSD4]

LGTM

[juliamagan]

LGTM