Additional E2E Tests for Vulnerability Detector

[Rebits]

Description

In the development of vulnerability detector E2E tests, we introduced new test cases to cover specific scenarios that were not part of the original plan.

This issue is created to track the development of these additional tests, which were not part of the initial plan. Additionally, the steps to manually reproduce these tests will be provided in this issue to facilitate progress for the Beta 2 Release.

Tests cases

• Installation of a vulnerable package when agent is offline
• Enable vulnerability detector when some agents are already registered
• Change agents' manager and install a vulnerable package

Tests Design

Installation of a vulnerable package when agent is offline

• Deploy a Wazuh cluster and two agents.
• Enable vulnerability detector in the managers.
• Register the agents to each manager host.
• Wait until feeds are updated.
• Wait until first scan is performed.
• Stop agents.
• Install a vulnerable package in both agents.
• Start agents.
• Wait until the next scan is performed.
• Check that the vulnerable package is detected in both agents

Enable vulnerability detector when some agents are already registered

• Deploy a Wazuh cluster and two agents.
• Disable vulnerability detector in the managers.
• Register the agents to each manager host.
• Enable vulnerability detector in the managers.
• Check that the agents are scanned and the vulnerable packages are detected.

Change agents' manager and install a vulnerable package

• Deploy a Wazuh cluster and two agents.
• Enable vulnerability detector in the managers.
• Register the agents to each manager host.
• Wait until feeds are updated.
• Wait until first scan is performed.
• Check that the agents are scanned and the vulnerable packages are detected.
• Change the manager of the agents.
• Install a vulnerable package in both agents.
• Wait until the next scan is performed.
• Check that the vulnerable package is detected in both agents

[Rebits]

Refactored TestInitialScans in order to include Enable vulnerability detector when some agents are already registered case

[Rebits]

• Finish Enable vulnerability detector when some agents are already registered case
• Working on Change agents' manager and install a vulnerable package case
• Included new check in all tests to detect unexpected warnings and errors in the environment
• Included debug and evidence gathering test options

[Rebits]

It has been detected a new error in vulnerability detection module. Expected alerts was not triggered. In addition only the mitigated alerts is different of the previously alerted vulnerability. Currently working in replicating this issue with debug option enabled:

{"timestamp":"2024-03-11T17:57:35.073+0000","rule":{"level":7,"description":"Yum package deleted.","id":"2934","firedtimes":3,"mail":false,"groups":["syslog","yum","config_changed"],"pci_dss":["10.6.1","10.2.7"],"gpg13":["4.10"],"gdpr":["IV_35.7.d"],"hipaa":["164.312.b"],"nist_800_53":["AU.6","AU.14"],"tsc":["CC7.2","CC7.3","CC6.8","CC8.1"]},"agent":{"id":"001","name":"ip-172-31-12-155.ec2.internal","ip":"172.31.12.155"},"manager":{"name":"ip-172-31-11-104"},"id":"1710179855.6984797","cluster":{"name":"wazuh","node":"manager2"},"full_log":"Mar 11 17:57:34 ip-172-31-12-155 yum[20897]: Erased: 1:java-1.6.0-openjdk-1.6.0.41-1.13.13.1.el7_3.x86_64","predecoder":{"program_name":"yum","timestamp":"Mar 11 17:57:34","hostname":"ip-172-31-12-155"},"decoder":{},"location":"/var/log/messages"}




{"timestamp":"2024-03-11T17:58:09.054+0000","rule":{"level":3,"description":"The CVE-2023-21939 that affected java-1.6.0-openjdk was solved due to a package removal/update or a system upgrade","id":"23502","firedtimes":3,"mail":false,"groups":["vulnerability-detector"],"gdpr":["IV_35.7.d"],"pci_dss":["11.2.1","11.2.3"],"tsc":["CC7.1","CC7.2"]},"agent":{"id":"001","name":"ip-172-31-12-155.ec2.internal","ip":"172.31.12.155"},"manager":{"name":"ip-172-31-11-104"},"id":"1710179889.6985232","cluster":{"name":"wazuh","node":"manager2"},"decoder":{"name":"json"},"data":{"vulnerability":{"cve":"CVE-2023-21939","cvss":{"cvss3":{"base_score":"5.300000"}},"enumeration":"CVE","package":{"architecture":"x86_64","name":"java-1.6.0-openjdk","version":"1:1.6.0.41-1.13.13.1.el7_3"},"published":"2023-04-18T20:15:14Z","reference":"https://lists.debian.org/debian-lts-announce/2023/09/msg00018.html, https://security.netapp.com/advisory/ntap-20230427-0008/, https://www.couchbase.com/alerts/, https://www.debian.org/security/2023/dsa-5430, https://www.debian.org/security/2023/dsa-5478, https://www.oracle.com/security-alerts/cpuapr2023.html","severity":"Medium","status":"Solved","title":"CVE-2023-21939 affecting java-1.6.0-openjdk was solved","type":"Packages","updated":"2023-11-08T23:07:27Z"}},"location":"vulnerability-detector"}



{"timestamp":"2024-03-11T17:58:45.149+0000","rule":{"level":7,"description":"New Yum package installed.","id":"2932","firedtimes":3,"mail":false,"groups":["syslog","yum","config_changed"],"pci_dss":["10.6.1","10.2.7"],"gpg13":["4.10"],"gdpr":["IV_35.7.d"],"hipaa":["164.312.b"],"nist_800_53":["AU.6","AU.14"],"tsc":["CC7.2","CC7.3","CC6.8","CC8.1"]},"agent":{"id":"001","name":"ip-172-31-12-155.ec2.internal","ip":"172.31.12.155"},"manager":{"name":"ip-172-31-11-104"},"id":"1710179925.6987140","cluster":{"name":"wazuh","node":"manager2"},"full_log":"Mar 11 17:58:45 ip-172-31-12-155 yum[20912]: Installed: 1:java-1.6.0-openjdk-1.6.0.41-1.13.13.1.el7_3.x86_64","predecoder":{"program_name":"yum","timestamp":"Mar 11 17:58:45","hostname":"ip-172-31-12-155"},"decoder":{},"location":"/var/log/messages"}
{"timestamp":"2024-03-11T17:59:11.921+0000","rule":{"level":10,"description":"CVE-2024-20952 affects java-1.6.0-openjdk","id":"23505","firedtimes":3,"mail":false,"groups":["vulnerability-detector"],"gdpr":["IV_35.7.d"],"pci_dss":["11.2.1","11.2.3"],"tsc":["CC7.1","CC7.2"]},"agent":{"id":"001","name":"ip-172-31-12-155.ec2.internal","ip":"172.31.12.155"},"manager":{"name":"ip-172-31-11-104"},"id":"1710179951.6987584","cluster":{"name":"wazuh","node":"manager2"},"decoder":{"name":"json"},"data":{"vulnerability":{"assigner":"oracle","cve":"CVE-2024-20952","cvss":{"cvss3":{"base_score":"7.400000","vector":{"availability":"NONE","confidentiality_impact":"HIGH","integrity_impact":"HIGH","privileges_required":"NONE","scope":"UNCHANGED","user_interaction":"NONE"}}},"enumeration":"CVE","package":{"architecture":"x86_64","condition":"Package default status","name":"java-1.6.0-openjdk","source":" ","version":"1:1.6.0.41-1.13.13.1.el7_3"},"published":"2024-01-16T22:15:42Z","rationale":"Vulnerability in the Oracle Java SE, Oracle GraalVM for JDK, Oracle GraalVM Enterprise Edition product of Oracle Java SE (component: Security).  Supported versions that are affected are Oracle Java SE: 8u391, 8u391-perf, 11.0.21, 17.0.9, 21.0.1; Oracle GraalVM for JDK: 17.0.9, 21.0.1; Oracle GraalVM Enterprise Edition: 20.3.12, 21.3.8 and  22.3.4. Difficult to exploit vulnerability allows unauthenticated attacker with network access via multiple protocols to compromise Oracle Java SE, Oracle GraalVM for JDK, Oracle GraalVM Enterprise Edition.  Successful attacks of this vulnerability can result in  unauthorized creation, deletion or modification access to critical data or all Oracle Java SE, Oracle GraalVM for JDK, Oracle GraalVM Enterprise Edition accessible data as well as  unauthorized access to critical data or complete access to all Oracle Java SE, Oracle GraalVM for JDK, Oracle GraalVM Enterprise Edition accessible data. Note: This vulnerability applies to Java deployments, typically in clients running sandboxed Java Web Start applications or sandboxed Java applets, that load and run untrusted code (e.g., code that comes from the internet) and rely on the Java sandbox for security. This vulnerability does not apply to Java deployments, typically in servers, that load and run only trusted code (e.g., code installed by an administrator). CVSS 3.1 Base Score 7.4 (Confidentiality and Integrity impacts).  CVSS Vector: (CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:N).","reference":"https://lists.debian.org/debian-lts-announce/2024/01/msg00023.html, https://www.oracle.com/security-alerts/cpujan2024.html, https://security.netapp.com/advisory/ntap-20240201-0002/","severity":"High","status":"Active","title":"CVE-2024-20952 affects java-1.6.0-openjdk","type":"Packages","updated":"2024-02-15T03:18:31Z"}},"location":"vulnerability-detector"}

[Rebits]

On hold in favor of Beta 4 testing

[Rebits]

A bug that affects Enable vulnerability detector when some agents are already registered and Change agents' manager and install a vulnerable package cases detected in #5103
Changing ETA accordingly to allow a fix libraries involved

---

• Created mocking for initial scan tests in order to allow fasten development
• Created Installation of a vulnerable package when agent is offline tests
• Improve errors in report HTML
• Improve evidence gathering logic
• Refactor tests classes

[Rebits]

• Fixed HTML report Vulnerability Detector's E2E test report is shifted #5109
• Working on fixing bugs in alerts and index gathering

[Rebits]

Several changes increasing performance and generated report:

• 9d1f2ac
• 4d73320
• 7133717
• 9d1f2ac

[Rebits]

• Working on change manager on agent test

[Rebits]

• Working on setup/teardown handling for install/remove/update package

[Rebits]

Unblocked. Continuing with #5266 development

[Rebits]

• test_install_vulnerable_package_when_agent_down completed
• test_change_agent_manager completed

---

Finish refactor of test_vulnerability_detector_scans_cases:

Report.zip

Some of the test cases fail:

• test_vulnerability_detector/test_vulnerability_detector.py::TestScanSyscollectorCases::test_vulnerability_detector_scans_cases[upgrade_package_nonvulnerable_to_nonvulnerable]::setup: Package information missing in the yaml
• test_vulnerability_detector/test_vulnerability_detector.py::TestScanSyscollectorCases::test_vulnerability_detector_scans_cases[install_package]: Expected alerts does not appear in the environment
• test_vulnerability_detector/test_vulnerability_detector.py::TestScanSyscollectorCases::test_vulnerability_detector_scans_cases[install_package_non_vulnerable]: Error installing the package

ERROR    root:remote_operations_handler.py:318 Error installing package on agent1: Failed to install package in agent1: {'changed': False, 'msg': 'A later version is already installed'}

Note

Test performed in a limited environment of 1 agent.

[Rebits]

Open PR Draft with additional changes: https://github.com/wazuh/wazuh-qa/pull/5287/files

Todo

• Refactor checks of test_change_agent_manager and test_vulnerability_detector_scans_cases to improve report readability
• Improve code readability of check_validators module
• Complete missing docstring
• Testing in a real environment

[Rebits]

Tests in the Jenkins pipeline have unexpectedly failed with no accompanying reports generated. Check here: https://ci.wazuh.info/job/Test_e2e_system/265/console

Investigating further; moving the issue to 'In Progress' to resolve

[Rebits]

Tests were launched again after PR requested changes:

• Assets1.zip
• Assets2.zip
• Report.zip

Where we can see the following errors:

• Database errors: Broken database during Vulnerability Detector tests wazuh#22847
• False positives: Issue pending to be opened
• Unexpected vulnerabilities: Issue pending to be opened
• No vulnerabilities were detected for agent4 in the vd_disabled_when_agents_registration test case. Issue pending to be opened.
• Duplicated vulnerabilities: Multiple vulnerabilities appear to be duplicated in the index, although we fail to replicate this issue manually or repeat the failed test cases

Further research is required