System Functionality Testing for Vulnerability Detector Feature

[davidjiglesias]

Overview

This issue is dedicated to the comprehensive end-to-end functionality system testing of the Vulnerability Detector feature. The aim is to ensure the correct operation of all interconnected components and processes involved in the Vulnerability Detector feature, with a focus on its alerting and state management capabilities. The test coverage spans across multiple operating systems, simulating real-world use and ensuring the robustness of the system across various scenarios.

Feature Architecture and Components

The Vulnerability Detector feature works through continuous system scanning and comparison against a vulnerability feed. It uses Wazuh's syscollector to gather software inventory, which is then cross-referenced with the feed.

The architecture includes:

1. Syscollector module: This is the input source that gathers the software inventory data from the Wazuh agent.
2. Vulnerability detector module: This module is the core of the feature, which takes the information from Syscollector and cross-references it with the vulnerability feed.
3. Vulnerability feed: This acts as a reference for known vulnerabilities. Any updates or modifications to this feed can trigger vulnerability detection.
4. State index: This component tracks the system's vulnerability statuses and alerts, with state changes being driven by events such as software installation, deletion, system scans, or updates to the vulnerability feed.
5. Alerts index: This manages alerts generated based on changes detected in the system state by the Wazuh Engine through the Vulnerability detector module. These alerts track status changes.

Test Design

The test design ensures that all components work as intended in an integrated, real-world context. We aim to ensure that the Vulnerability Detector feature behaves reliably, issuing appropriate alerts and maintaining accurate state information across various scenarios.

TBD

Chosen Families

• Windows
• MacOS
• Redhat based
• Debian based

Initial Coverage OS

• Windows 11
• Windows Server 2022
• MacOS Ventura or Sonoma (Latest available at tests delivery)
• CentOS 7
• Ubuntu 22.04

This list will be updated accordingly following the new compatibility matrix and tiers system.

Test Cases

Trigger/Condition	Preconditions	Expected Outcome	Type
First syscollector scan (Rsync)	TBD	System states and initial vulnerability alerts recorded ("added" only)	Event driven
Subsequent scan (Dbsync) without any installation	TBD	System states and alerts remain unchanged	Time driven
Installation of a vulnerable package	TBD	New entry appears in system states and a vulnerability "added" alert triggered	Time driven
Installation of a non-vulnerable package	TBD	System states and alerts remain unchanged	Time driven
Updating the vulnerability feed with a new applicable vulnerability	TBD	New entry appears in system states and a vulnerability "added" alert is triggered	Event driven
Updating the vulnerability feed with a new non-applicable vulnerability	TBD	System states and alerts remain unchanged	Event driven
Deleting the states index	TBD	The state index needs to be regenerated	Event driven
Deleting a non-vulnerable package	TBD	System states and alerts remain unchanged	Time driven
Deleting a vulnerable package	TBD	Entry of the vulnerable package disappears from the state and a "fixed" alert is triggered	Time driven
Updating the vulnerability feed removes a CVE and causes agent to cease to be vulnerable	TBD	The applicable CVE disappears from the state and a "fixed" alert is triggered	Event driven
Updating a vulnerable package that remains vulnerable to the same CVE	TBD	The version changes in the states index and two alerts are generated (one "fixed" and another "added")	Time driven
Updating a vulnerable package that becomes vulnerable to another CVE	TBD	The version and the CVE change in the states index and two alerts are generated (one "fixed" and another "added")	Time driven
Updating a vulnerable package that becomes vulnerable to another CVE and retains the previous one	TBD	The version changes for the existing CVE and a new CVE appears in the states index, with three alerts generated (one "fixed" and two "added")	Time driven
Updating a vulnerable package that ceases to be vulnerable	TBD	The entry for the package disappears from the states index and a "fixed" alert is triggered	Time driven
Updating a non-vulnerable package that becomes vulnerable	TBD	A new state appears in the states index and an "added" alert is triggered	Time driven
Updating a non-vulnerable package that remains non-vulnerable	TBD	System states and alerts remain unchanged	Time driven

Test Execution

Security Implications:

• TBD

Performance Expectations:

• TBD

Edge Cases/Exception Cases:

• TBD

Regression Scenarios:

• TBD

Tasks

Testing environment

• https://github.com/wazuh/wazuh-jenkins/issues/5660
• https://github.com/wazuh/wazuh-jenkins/issues/5661
• https://github.com/wazuh/wazuh-jenkins/issues/5662
• https://github.com/wazuh/wazuh-jenkins/issues/5228
• https://github.com/wazuh/wazuh-jenkins/issues/4641
• Migrate Wazuh Ansible QA custom roles #4581
• Designing a Comprehensive Testing Environment for Vulnerability Detector Feature Testing #4582
• https://github.com/wazuh/wazuh-jenkins/issues/5774
• https://github.com/wazuh/wazuh-jenkins/issues/5883
• https://github.com/wazuh/wazuh-jenkins/issues/5925
• https://github.com/wazuh/wazuh-jenkins/issues/5237
• https://github.com/wazuh/wazuh-jenkins/issues/5949
• Add Ventura MacOS support to End to End tests pipeline #4737
• https://github.com/wazuh/wazuh-jenkins/issues/6041

Framework

• Research required tools for performing VD tests #4587
• Research implementation of dependency and parallel tests in the current QA framework #4588

Testing requirements

• Determine VDT System tests tiers, preconditions, implications and expectations #4531
• Research vulnerable packages for usage in VDT System tests #4529
• Prepare vulnerability feeds for VDT System tests #4530

Tests Development

• Add Vulnerability Detector System End-to-End syscollector initial scans tests #4590 - Should be completed as first part of test development
• Add Vulnerability Detector System End-to-End tests #4591

CI

• Include System Functionality tests into the CI processes #4589

[Rebits]

We need to expand the requirements because it's currently unclear which architecture the tests should be conducted on. For more details, please refer to the following link: https://github.com/wazuh/wazuh-jenkins/issues/5774.

Furthermore, the precise infrastructure in which the tests should be carried out remains unclear. We need additional information, such as whether it should be on a single node, a multi-node manager setup, an indexer multinode configuration, or another specific setup. To proceed, we require further details. More information and a infrastructure suggestion can be found in #4582

Finally, is necessary to specify the list of the CI process where this tests will be included. This is necessary to estimate the #4589 task

[Rebits]

Meeting with @davidjiglesias confirming the following points related to the infrastrucutre:

• Testing environment should include a cluster, at least with 1 worker.
• Both architectures, x86-64 and arm should be supported.
• CI process will be determinate in the future.

[Rebits]

In today's meeting, attended by @Deblintrake09, @juliamagan, @davidjiglesias, and @Rebits, a resolution was reached under the guidance of @davidjiglesias. It has been decided to forego support for macOS Sonoma in our testing environments, citing complications in its integration. Instead, our focus will shift to supporting macOS Ventura ARM.

The details of this decision and the transition plan will be documented in the corresponding issue for further reference and implementation: #4737

[Rebits]

Adjusted ETA to December 20, 2023, with approval from @davidjiglesias.

This modification is prompted by a development delay now anticipated for December 18: wazuh/wazuh#14153

[Rebits]

Testing cannot proceed at the moment due to functionality issues identified in wazuh/wazuh#20785. We will resume development once these issues are addressed.

[Rebits]

A segmentation fault has been identified in the current environment status

• Segmentation fault in Vulnerability Detector Refactor wazuh#21176

Unfortunately, testing cannot proceed until a stable version is delivered.

---

We have decided to exclude https://github.com/wazuh/internal-devel-requests/issues/472 from the development. The current approach involves utilizing public IP addresses instead, as it helps prevent errors in VPN handling.
More information in https://github.com/wazuh/wazuh-jenkins/issues/6041

[Rebits]

wazuh/wazuh#21176 seems to be solved. We can proceed with the development of the tests

[davidjiglesias]

LGTM

[Rebits]

Missing tests cases included in the middle of the development will be tracked in #4914