qualys_vmdr: expand CVE list for each vulnerability

[efd6]

Proposed commit message

Reference for the DTD: https://cdn2.qualys.com/docs/qualys-api-vmpc-xml-dtd-reference.pdf

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Fixes Missing CVE list in the Knowledge Base integration #9364

Screenshots

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[clement-fouque]

I think we can keep it to something simpler like an array of all CVEs. If we have one document per CVE per QID, it'll make enrichment harder (as it'll match multiple documents). By keeping an array of CVEs, we can easily enrich on any CVE in the array.

I think the final document should be similar to the following:

[
    {
        "qualys_vmdr": {
            "knowledge_base": {
                "title": "Amazon Linux Security Advisory for Hypertext Preprocessor (PHP) : ALAS2PHP8.0-2023-005",
                "qid": "356044",
                "cve_list": [
                    "CVE-2022-31629",
                    "CVE-2022-31628"
                ]
            }
        }
    },
    {
        "qualys_vmdr": {
            "knowledge_base": {
                "title": "Another vulnerability",
                "qid": "111111",
                "cve_list": [
                    "CVE-2024-00001",
                    "CVE-2024-00002"
                ]
            }
        }
    }
]

[efd6]

@clement-fouque So, you basically just want to drop the URLs and remove the then-redundant object?

[clement-fouque]

@clement-fouque So, you basically just want to drop the URLs and remove the then-redundant object?

Yes, correct (if by then-redundant object you mean one document per CVE per QID).

[efd6]

if by then-redundant object you mean one document per CVE per QID

Not quite; from cve_list: [{id, url}, ...] to cve_list: [id, ...].

[efd6]

Also, I'm trying to understand how this would be consistent with ECS. As noted in the issue this would result in an array in vulnerability.id, but this is not noted to be an array.

[efd6]

This is arguably now not a breaking change. The field definitions are different to how they were before, but they did not work previously.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 4aedaab

History

• 💚 Build #9653 succeeded e04e210
• 💚 Build #9651 succeeded 8a820b1f0bfe0b33a7ad8fc104248460d1198385

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
83.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[clement-fouque]

Also, I'm trying to understand how this would be consistent with ECS. As noted #9364 (comment) this would result in an array in vulnerability.id, but this is not noted to be an array.

I'm not sure I understand why it'll result in an array in vulnerability.id (I got the array part but not why vulnerability.id will be filled in). For the Knowledge Base datastream, I don't think we need (and we have) to copy any value into vulnerability.id.

I think it's good to only have the array field qualys_vmdr.knowledge_base.cve_list.

The only element that I think of to have one document per CVE per QID is if we want to enrich other sources (e.g. Snyk, Rapid7) with the Qualys Knowledge Base. It is unlikely, but still I think it would work if we define the match field on the multi-value field qualys_vmdr.knowledge_base.cve_list.

[efd6]

The previous code attempted to set vulnerability.id from the qualys_vmdr.knowledge_base.cve_list. If this is not wanted, it can be remove and would solve the issue. However, the description of vulnerability.id does suggest that the CVE ID should go there.

[clement-fouque]

I think it's better to comply with ECS and don't copy CVE IDs into the vulnerability.id field. Users will have the possibility to do it on their own.

[kcreddy]

We do have few packages that are using vulnerability.id and vulnerability.reference as an array even though ECS doesn't state them explicitly.
Examples;
https://github.com/elastic/integrations/blob/main/packages/tenable_io/data_stream/plugin/_dev/test/pipeline/test-plugin.log-expected.json#L124-L130
https://github.com/elastic/integrations/blob/main/packages/amazon_security_lake/data_stream/event/_dev/test/pipeline/test-findings.log-expected.json#L204-L210

Since Elasticsearch supports array out-of-the-box, maybe we could add multiple CVEs into vulnerability.id.

[kcreddy]

LGTM

[elasticmachine]

Package qualys_vmdr - 3.0.0 containing this change is available at https://epr.elastic.co/search?package=qualys_vmdr