Create VD offline efficacy tests to validate scanner behavior #28210

pereyra-m · 2025-02-13T13:22:40Z

Description

The efficacy tests were moved at:

Migrate the vulnerability detector efficacy tests workflow for 4.x #27943

But we've found that creating the offline tests in that issue would be too much work.
We've decided to split the requirements.

DoD

Create an offline test feed to validate all the VD use cases
Create efficacy tests that don't depend on the content changes
Make sure that all the possible use cases are covered (OS, packages, alerts, etc.)

MiguelazoDS · 2025-02-25T19:27:14Z

Update

To be discussed with the team, but I think the best approach is to use the latest snapshot at this moment (vd_1.0.0_vd_4.8.0_1317226_1740423267.json) and create a database from there, and then restore most of the test we already had that were removed as part of the requirements here #28106.

With that we ensure there's no changes in the content that could break the tests, and we keep a real content, without the need to mock package names and create custom CVEs.

Considering the size gap between the snapshot and the generated database is not too much, and I don't think we would change those test often, upload a compressed .tar.xz in the qa tests folder may be an option since generating a database with the snapshot will take so much time.

But we need to be able to change the content easily if we, for instance, need to add some tests for CISA because the snapshot may be no longer available to generate a new database.

Proposal

With that in mind, I would modify the snapshot previously mentioned to keep only the CVEs that make the tests pass, so we reduce the size of the snapshot and speed up the database generation process. I don't see practical to compress the snapshot file, since we need to detect changes easily to trigger a new workflow. The current size is aprox 2.7GB.

As mentioned, I would generate a offline compressed database on demand and during offline snapshot changes, then upload to s3 as we do with the official content database. Then we can download from there and maintain pretty much the same code and test real operating systems and packages.

To be evaluated: If the snapshot is too small, we may be able to generate a database on each PR without the need to create another workflow to generate the offline database in main branch and use s3 to store it.

MiguelazoDS · 2025-02-27T14:34:23Z

Update

Discussing the previous discrepancy with @pereyra-m we found that the msg differ due to the inventory being empty in the case of a reduced content. Once we added an extra CVE to avoid that behavior the scanner executes the cleanAgentInventory method.

Now I'll start to add new checks to the logs and documentation.

MiguelazoDS · 2025-02-27T20:52:38Z

Findings

The implementation is done but I found some behaviors that need revision

The first total harmless, I see a reduntant log. Really not that important

    "Scanning OS - 'macos' (Installed Version: 14.0, Security Vulnerability: CVE-2024-23224). Identified vulnerability: Version: 13.0. Required Version Threshold: 13.6.4. Required Version Threshold (or Equal): .",
    "Scanning OS - 'macos' (Installed Version: 14.0, Security Vulnerability: CVE-2024-23224). Identified vulnerability: Version: 14.0. Required Version Threshold: 14.3. Required Version Threshold (or Equal): .",

Active OS vulnerability alerts are not triggered.

Note

It seems is deliberately implemented like that using the !context->m_isFirstScan.

MiguelazoDS · 2025-02-28T20:52:14Z

Update

Important

Found the issue mentioned above. I could obtain the descriptive information from NVD in a local environment.
We can see that the database is small, and it is the result of generating it with the reduced offline feed.

Important

The cause of this is the different compilation method. For some reason compiling the whole project make TARGET=server does not work as expected.

root@jammy:/home/vagrant/wazuh/src# du -sh queue/
52M  queue/
root@jammy:/home/vagrant/wazuh/src# wazuh_modules/vulnerability_scanner/build/testtool/rocksDBQuery/rocks_db_query_testtool -d queue/vd/feed -c descriptions_nvd -f wazuh_modules/vulnerability_scanner/schemas/vulnerabilityDescription.fbs | grep -A 20 "CVE-2014-3524"
CVE-2014-3524 ==> {
  "accessComplexity": "MEDIUM",
  "assignerShortName": "redhat",
  "attackVector": "",
  "authentication": "NONE",
  "availabilityImpact": "COMPLETE",
  "classification": "CVSS",
  "confidentialityImpact": "COMPLETE",
  "cweId": "CWE-77",
  "datePublished": "2014-08-26T14:55:05Z",
  "dateUpdated": "2022-02-07T16:25:26Z",
  "description": "Apache OpenOffice before 4.1.1 allows remote attackers to execute arbitrary commands and possibly have other unspecified impact via a crafted Calc spreadsheet.",
  "integrityImpact": "COMPLETE",
  "privilegesRequired": "",
  "reference": "http://secunia.com/advisories/59600, http://secunia.com/advisories/59877, http://secunia.com/advisories/60235, http://www.securityfocus.com/archive/1/533200/100/0/threaded, http://www.securityfocus.com/bid/69351, http://www.securitytracker.com/id/1030755, https://security.gentoo.org/glsa/201603-05, https://exchange.xforce.ibmcloud.com/vulnerabilities/95421, http://blog.documentfoundation.org/2014/08/28/libreoffice-4-3-1-fresh-announced/, http://www.openoffice.org/security/cves/CVE-2014-3524.html",
  "scope": "",
  "scoreBase": 9.3,
  "scoreVersion": "2.0",
  "severity": "HIGH",
  "userInteraction": ""
}

Note

This does not rely on the compilation method itself, but due to some non-clean environment. It could not be reproduced by the team.

MiguelazoDS · 2025-03-02T18:41:35Z

Update

Failing test

https://github.com/wazuh/wazuh/actions/runs/13608675868/job/38043180118?pr=28390

Note

It passes after re-launch. It's a flaky test an issue will be created

Missing descriptions_debian key

It seems that this CVE does not have Debian descriptive information CVE-2023-34966, and when it's the only CVE in the content it triggers the exception avoiding continue the normal flow. This won't occur with a full content, but probably needs revision. So for these test, it was added the CVE-2024-3094

Wrong CVE type

When a hotfix event arrives, we're looking for packages vulnerabilities without considering os vulnerabilities.

We should consider both. In the first line the CVE was cleaned because of the KB and the change in the cveSolvedInventory class, but the second line is not evaluated even having the right KB.

root@jammy:/home/vagrant/wazuh/src# ../../rocksdb/build/tools/ldb --db=queue/vd/inventory scan --column_family=os
002_Microsoft Windows 10 ==> CVE-2024-20659
root@jammy:/home/vagrant/wazuh/src# ../../rocksdb/build/tools/ldb --db=queue/vd/inventory scan --column_family=package
002_9ca216dec5bef19993deb9d365debf11e7f554f9 ==> CVE-2016-0145

Confusing log message

If we have more than one CVE related to an OS or Package, the log messages changes from this

Deleting agent element key: 002_Microsoft Windows 10

to this

Updating agent element key: 002_Microsoft Windows 10 -> CVE-2024-20659

Without notifying what CVE was solved.

Skype translation collision

Adding this package

{
            "architecture": "i686",
            "checksum": "72cb38e06710a81ba437ad222e3a6aaeefe90352",
            "description": " ",
            "format": "win",
            "item_id": "9ca216dec5bef19993deb9d365debf11e7f554f9",
            "multiarch": null,
            "name": "Skype for Business Basic 2016",
            "priority": " ",
            "scan_time": "2025/03/05 14:04:53",
            "size": 0,
            "source": " ",
            "vendor": "Microsoft Corporation",
            "version": "16.0.4849.1000"
        }

I found the following log messages

wazuh-modulesd:vulnerability-scanner:packageScanner.hpp:145 scanPackageTranslation : Translation for package 'Skype for Business Basic 2016' in platform 'windows' found in Level 2 cache.
wazuh-modulesd:vulnerability-scanner:packageScanner.hpp:169 scanPackageTranslation : Initiating a vulnerability scan for package 'skype' (win) (microsoft) with CVE Numbering Authorities (CNA) 'nvd' on Agent '' (ID: '002', Version: '').
wazuh-modulesd:vulnerability-scanner:packageScanner.hpp:169 scanPackageTranslation : Initiating a vulnerability scan for package 'skype' (win) (skype) with CVE Numbering Authorities (CNA) 'nvd' on Agent '' (ID: '002', Version: '').
wazuh-modulesd:vulnerability-scanner:packageScanner.hpp:169 scanPackageTranslation : Initiating a vulnerability scan for package 'skype' (win) (skype_technologies) with CVE Numbering Authorities (CNA) 'nvd' on Agent '' (ID: '002', Version: '').
wazuh-modulesd:vulnerability-scanner:packageScanner.hpp:776 handleRequest : Vulnerability scan for package 'Skype for Business Basic 2016' on Agent '002' has completed.

This tells me that the translation for Skype for business package is not working because is matching with the Skype package.

Note

This offline test was modified manually to make the test pass, the translation not only collides but also it does not match the vendor. A new issue will be created to address this.

MiguelazoDS · 2025-03-12T16:24:27Z

Update

Minor update, I forgot to add the expected for Skype For Business pkg. The issue with the translation was notified to the team in charge.

pereyra-m added level/task type/test labels Feb 13, 2025

wazuhci moved this to Backlog in XDR+SIEM/Release 4.12.0 Feb 13, 2025

wazuhci added this to XDR+SIEM/Release 4.12.0 Feb 13, 2025

fdalmaup added the request/operational label Feb 14, 2025

wazuhci removed this from XDR+SIEM/Release 4.12.0 Feb 14, 2025

pereyra-m assigned MiguelazoDS Feb 24, 2025

MiguelazoDS linked a pull request Feb 25, 2025 that will close this issue

Offline feed efficacy vd tests #28390

Open

This comment has been minimized.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create VD offline efficacy tests to validate scanner behavior #28210

Create VD offline efficacy tests to validate scanner behavior #28210

pereyra-m commented Feb 13, 2025 •

edited by MiguelazoDS

Loading

MiguelazoDS commented Feb 25, 2025 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

MiguelazoDS commented Feb 27, 2025

MiguelazoDS commented Feb 27, 2025 •

edited

Loading

This comment has been minimized.

MiguelazoDS commented Feb 28, 2025 •

edited

Loading

MiguelazoDS commented Mar 2, 2025 •

edited

Loading

MiguelazoDS commented Mar 12, 2025

Create VD offline efficacy tests to validate scanner behavior #28210

Create VD offline efficacy tests to validate scanner behavior #28210

Comments

pereyra-m commented Feb 13, 2025 • edited by MiguelazoDS Loading

Description

DoD

MiguelazoDS commented Feb 25, 2025 • edited Loading

Update

Proposal

This comment has been minimized.

This comment has been minimized.

MiguelazoDS commented Feb 27, 2025

Update

MiguelazoDS commented Feb 27, 2025 • edited Loading

Findings

This comment has been minimized.

MiguelazoDS commented Feb 28, 2025 • edited Loading

Update

MiguelazoDS commented Mar 2, 2025 • edited Loading

Update

Failing test

Missing descriptions_debian key

Wrong CVE type

Confusing log message

Skype translation collision

MiguelazoDS commented Mar 12, 2025

Update

pereyra-m commented Feb 13, 2025 •

edited by MiguelazoDS

Loading

MiguelazoDS commented Feb 25, 2025 •

edited

Loading

MiguelazoDS commented Feb 27, 2025 •

edited

Loading

MiguelazoDS commented Feb 28, 2025 •

edited

Loading

MiguelazoDS commented Mar 2, 2025 •

edited

Loading