Integration tests: Reliable full yellow local for 4.2.0

[juliamagan]

Wazuh QA: Branch	Wazuh QA: Commit	Wazuh: Tag	Wazuh: Commit
4.2	a043e36	v4.2.0-rc8	wazuh/wazuh@a594dcd

It is necessary to get a reliable full yellow on the integration tests of the wazuh-qa repository. We will need to do research to check the status of the integration tests. If any problem is found during the research, we will need to fix it in order to achieve the reliable full green.

Fixing

---

Research

• Research: Check the integration tests status in 4.2.0-RC8 #1493

Integration tests fixing

• Full Yellow Local - Fix Rootcheck tests in 4.2 #1520
• Full Yellow Local - Fix Agentd tests in 4.2 #1521
• Full Yellow Local - Fix WPK tests in 4.2 #1528
• Full Yellow Local - Fix Remoted tests in 4.2 #1530
• Full Yellow Local - Fix Vulnerability Detector tests in 4.2 #1531
• Full Yellow Local - Fix GCloud tests in 4.2 #1532
• Full Yellow Local - WazuhDB tests in 4.2 #1535
• Full Yellow Local - Delete Cluster tests in 4.2 #1538
• Full Yellow Local - Fix FIM tests in 4.2 #1539

QA Framework shared libraries fixing

• Full Yellow Local - Fix Agent simulator Rootcheck class in 4.2 #1580

Report

---

Product issues reported

• Database is locked using Vulnerability Detector wazuh#9309 (Affecting Vulnerability detector tests)
• NET command on Windows times out when stopping the agent wazuh#9298 (Affecting Agentd tests)
• Windows agent does not show error messages in ossec.log when applying invalid internal Wazuh configuration wazuh#8746 (Affecting Agentd tests)

QA framework issues reported

• Refactor FileMonitor because of inconsistent behavior when heavy log activity #1602
• Research and Refactor Agent Simulator for Remoted test #1698

QA framework enhancement propositions

• Modify the QA framework to the save the logs after every test #1690

PRs merged

• Remoted

  • Replacing the MITM usage by the archives.log monitor in the test_multi_agent_protocols_communication tests #1692
  • Shutting down the agent socket before closing to avoid malformed messages errors #1674
  • Incrementing the time to wait for active status in order to handle the case of having a pending status #1662
  • Avoid updating the position of FileMonitor when monitoring multiples logs in remoted tests #1653
  • Add timeout for the manager services to be ready in test_active_response_send_ar.py #1634
  • Increasing wait time until socket monitor is fully initialized in test_multi_agent_protocols_communication.py #1625
  • Add timeout for remoted socket to be ready in test_request_agent_info.py #1624
  • Disable inconsitent remoted tests #1699

• Vulnerability Detector

  • Enable V.D. at startup to create the CVE database in test_general_settings_ignore_time.py #1650
  • Apply correct formatting to wazuh_providers_multiple_providers.yaml #1647
  • Fix and optimize: test_cpe_indexing.py #1656
  • Timeout Extended to Vulnerability Detector Tests #1687
  • Clear log file before run each use case in test_invalid_values_redhat_feed.py #1631
  • Clear log file before run each use case in test_missing_fields_redhat_feed.py #1633
  • Clear log file before run each use case in test_invalid_syntax_redhat_feed.py #1630
  • Change year of feed update in test_download_feeds.py #1617
  • Disabling unused modules to improve stability of the vulnerability detector module integration tests #1607
  • Replacing custom NVD feed because vulnerabilities are not being inserted #1603
  • Increasing timeout for vulnerability detector tests that require many files downloads #1604

• Agentd

  • Fix test_agentd_state_config.py for Windows #1592
  • Fix: Delete Auto Negotation TLSv1_1 test case #1629
  • Full Yellow Local - Fix Agentd tests in 4.2 #1702

• GCP

  • Full Yellow Local - Fix GCP tests in 4.2 #1621

• FIM

  • Removed test_response_timeout.py #1752
  • Fix test_registry/test_registry_basic_usage/test_long_registry_path.py #1669
  • Fix test_fim/test_files/test_prefilter_cmd.py #1597
  • Fix test_prefilter_cmd_conf in scheduled mode #1646
  • Improve validation of registry entries #1670
  • Fix test test_fim/test_files/test_skip/test_skip.py #1661
  • Fix test_files/test_max_files_per_second/test_max_files_per_second.py #1667

Tests disabled

• Remoted:
  Affected by Research and Refactor Agent Simulator for Remoted test #1698

  • test_active_response_send_ar/test_active_response_send_ar.py
  • test_multi_agent_status
  • test_agent_version_shared_configuration_startup_message

• Vulnerability Detector:
  Affected by Refactor FileMonitor because of inconsistent behavior when heavy log activity #1602

  • test_feeds/test_download_feeds
  • test_feeds/test_invalid_type_custom_feeds
  • test_providers/test_providers_enabled
  • test_providers/test_providers_no_os
  • test_providers/test_providers_update_from_year
  • test_scan_results/test_macos_inventory
  • test_scan_results/test_msu_inventory_msu_feed
  • test_scan_results/test_scan_providers_and_nvd_feed
  • test_windows/test_cpe_indexing
  • test_scan_results/test_scan_different_cves
  • test_scan_results/test_scan_nvd_feed
  • test_scan_results/test_redhat_duplicate_vulns

  Affected by Database is locked using Vulnerability Detector wazuh#9309

  • test_scan_results/test_ubuntu_inventory_canonical_feed
  • test_scan_results/test_redhat_inventory_redhat_feed
  • test_scan_results/test_debian_inventory_debian_feed
  • test_providers/test_providers_os
  • test_feeds/test_invalid_type_url_feeds

• Agentd

  • Test case: Auto Negotation TLSv1_1 (Fix: Delete Auto Negotation TLSv1_1 test case #1629)

• FIM
  Affected by NET command on Windows times out when stopping the agent wazuh#9298

  • test_fim/test_synchronization/test_response_timeout.py

Metrics

---

Check below some big numbers regarding the teamwork done here.

WPK
22 total comments
245 lines added, 74 lines removed
15 test rounds

Agentd
12 total comments
36 lines added, 29 lines removed
15 test rounds

GCP
6 total comments
227 lines added, 250 lines removed
9 test rounds

Remoted
30~ comments
77 lines added, 42 lines removed
128 tests rounds

WazuhDB
4~ comments
44 lines added, 5 lines removed
10 tests rounds

Vulnerability Detector
50~ comments
2,574 lines added, 595 lines removed
225 tests rounds

FIM
30~ comments
425 lines added, 666 lines removed
120 tests rounds

Conclusions

---

The majority of current integration tests were broken mostly because either the framework is not working properly or the host set up to run the tests was not correct.

There are big problems related to the amount of time that a test needs to finish. Major problems as well with the performance of internal functions like FileMonitor or ManInTheMiddle that we must fix for next iterations.

The full yellow local 4.2.0 has established a set of rules regarding new tests creation (for example, no more simulators) and IMO is the first step to achieve a solid and reliable testing framework by the end of this year.

[snaow]

I am taking out the following two issues:

• https://github.com/wazuh/wazuh-jenkins/issues/2649
• https://github.com/wazuh/wazuh-jenkins/issues/2648

Since they are related to Jenkins pipelines and this epic has turned into a full local testing effort.

[damarisg]

Working on Full Yellow Local - Fix Vulnerability Detector tests in 4.2 was created some issues:

• Database is locked using Vulnerability Detector wazuh#9309
• Refactor FileMonitor because of inconsistent behavior when heavy log activity #1602

Working on Full Yellow Local - Fix Remoted tests in 4.2 an issue was created to apply some improvements:

• Modify the QA framework to the save the logs after every test #1690

[snaow]

Closed by: #1843