Is comparing an Ubuntu report to a Debian report an apples-to-apples comparison?

[gordonmessmer]

I'm not sure how to phrase the subject.

If I scan similarly configured Debian 12 and Ubuntu 24.04 systems, I get fewer known CVEs on the Debian system. That might mean that Debian is fixing more vulnerabilities, or it might mean that Canonical is confirming more CVE reports.

Can results be compared between these two, or should readers treat those distributions as products that report vulnerabilities differently and therefore cannot compare results between each other?

(This post is a copy of google/osv.dev#6068 which I had meant to start here... I think. I'm actually not sure which project is correct.)

[BenkiNew]

Raw finding counts alone are not an apples-to-apples measure of distribution security, even when the machines are similarly configured. A lower count does not by itself establish either faster fixing or less complete reporting.

One concrete reason is that OSV's data sources differ: its data documentation lists a conversion pipeline for Debian Security Advisories, while Ubuntu is listed among current data sources through Canonical's ubuntu-security-notices repository. Neither count should be assumed to represent an identical census of every issue in the distribution's tracker.

For a useful comparison, I would start with matched package/CVE examples rather than totals:

• Keep scanner version, scan method, configuration, scan time and installed-package inventory comparable.
• Compare the same CVE and corresponding source package, using the full distribution package version and release.
• Check each distribution's affected/fixed status in the Debian tracker and Ubuntu CVE tracker, then check the corresponding OSV records and aliases. Avoid counting multiple advisory IDs for the same CVE as separate vulnerabilities.

Could you share the scanner version, command and a few CVEs present only in the Ubuntu result, with their package names, versions and ecosystem identifiers? That would distinguish a real package-status difference from inventory, matching or advisory-coverage differences. Without those inputs, the cause of your particular count difference remains undetermined.

[gordonmessmer]

One concrete reason is that OSV's data sources differ: its data documentation

That's very interesting, thank you!

Check each distribution's affected/fixed status in the Debian tracker and Ubuntu CVE tracker

Yeah, that seems like the next step. I did a couple of spot checks, and initially it seems like Ubuntu vulnerabilities are over-reported and Debian vulnerabilities are under-reported.

Could you share the scanner version, command and a few CVEs present only in the Ubuntu result

The scripts that created the images I scanned are here: https://github.com/gordonmessmer/security-scans

And the resulting scans are here: https://drive.google.com/file/d/1Tbk0pyBP6xi1lMOvidE5PBUufjjIap60/view?usp=sharing

[another-rex]

Just going by the number of CVEs is definitely not accurate. The primary use case of each report is listing the packages you should update to fix (and ideally show the severity so you can ignore the ones that don't really affect you).

The canonical records are more "official", as we get them straight from Canonical themselves, while the debian records we convert from the DSA and Debian security tracker.

Can results be compared between these two, or should readers treat those distributions as products that report vulnerabilities differently and therefore cannot compare results between each other?

I think the answer is that you cannot compare those two against each other.

[gordonmessmer]

The primary use case of each report is listing the packages you should update to fix

I'm scanning fully patched hosts. I interpret the positive results as confirmation of vulnerabilities that haven't been fixed yet, which is expected.

The canonical records are more "official", as we get them straight from Canonical themselves, while the debian records we convert from the DSA and Debian security tracker.

That makes sense. Thanks.