Transition Fleet from OVAL to OSV feeds for Ubuntu vulnerabilities

[getvictor]

Goal

User story
As a Fleet administrator managing Linux hosts,
I want vulnerability detection to use per-CVE OSV data instead of advisory-grouped OVAL data
so that I can trust that each CVE is correctly attributed to the specific packages it actually affects.

Context

Fleet currently uses OVAL feeds for vulnerability scanning on Ubuntu, Debian, and RHEL hosts (plus goval-dictionary for Amazon Linux). These feeds have structural limitations that cause false positives, particularly the Ubuntu USN OVAL feed, which groups all CVEs from a single Ubuntu Security Notice into one definition.

The OSV format (Open Source Vulnerabilities) is a newer, industry-standard JSON schema that provides per-CVE, per-release, per-package granularity. Multiple Linux distributions now publish native OSV data, making it a viable replacement for OVAL across most of Fleet's Linux vulnerability scanning.

The false positive problem (motivating example)

Example: #39370

USN-7027-1 fixes 9 CVEs across emacs, xemacs21, and org-mode. The USN OVAL definition uses a single test that checks ALL emacs binary packages (including emacs-common), so all 9 CVEs get attributed to emacs-common — even though CVE-2024-30205 only affects org-mode and the USN description says it doesn't apply to Ubuntu 24.04 at all.

The OSV CVE record for CVE-2024-30205 correctly lists only xemacs21, xemacs21-packages, and org-mode as affected on Noble. emacs/emacs-common are not listed.

Current OVAL sources and OSV availability

Platform	Current source	OSV available?	OSV ecosystem	Notes
Ubuntu 14.04–25.04	USN OVAL	Yes, native (June 2024)	Ubuntu	Per-CVE records with binary package names
Debian 9–12	Debian OVAL	Yes, converted from Debian Security Tracker	Debian	Includes unfixed CVEs, not just DSAs
RHEL 6–9	RHEL OVAL	Yes, native (Nov 2024)	Red Hat	RPM content; container support planned
Amazon Linux 2	RHEL 7 OVAL (proxy)	No	—	Currently uses wrong distro's OVAL
Amazon Linux 2022/2023	goval-dictionary	No	—	Keep goval-dictionary for now

Why OVAL was chosen originally

The USN OVAL feed was adopted in PR #6102 (June 2022) without documented comparison to alternatives. At the time, OVAL was the standard machine-readable format for Linux vulnerability data. OSV didn't exist for Ubuntu until June 2024, for Debian until mid-2024, and for RHEL until November 2024.

Current OVAL limitations beyond the grouping issue

• extend_definition XML elements are not parsed, silently dropping OS version checks from criteria trees
• The DefinitionXML struct doesn't differentiate between patch and vulnerability class definitions
• Matching rules in matching_rules.go are a manual band-aid for known false positives
• Amazon Linux 2 uses RHEL 7's OVAL file as a proxy, which is inherently inaccurate

OSV advantages

• Per-CVE records (not grouped by advisory)
• Per-release affected package lists (knows which CVEs affect which distro versions)
• Binary package names included (maps source packages to their installed binaries)
• Fixed version per package per release
• Unified schema across distros (same parser for Ubuntu, Debian, RHEL)
• Standard JSON is simpler to parse than OVAL XML with its criteria trees, extend_definitions, and variable references
• Available via osv.dev API, GCS bucket ( https://storage.googleapis.com/osv-vulnerabilities/<ECOSYSTEM>/all.zip), and per-distro GitHub repos

Changes

Product

• UI changes: No changes
• CLI (fleetctl) usage changes: No changes
• YAML changes: No changes
• REST API changes: No changes
• Fleet's agent (fleetd) changes: No changes
• Fleet server configuration changes: No changes
• Exposed, public API endpoint changes: No changes
• fleetdm.com changes: No changes
• GitOps mode UI changes: No changes
• GitOps generation changes: No changes
• Activity changes: No changes
• Permissions changes: No changes
• Changes to paid features or tiers: No changes
• My device and fleetdm.com/better changes: No changes
• Usage statistics: No changes
• Other reference documentation changes: No changes
• First draft of test plan added
• Once shipped, requester has been notified
• Once shipped, dogfooding issue has been filed

Engineering

Proposed implementation approach:

Library: https://pkg.go.dev/github.com/ossf/osv-schema/bindings/go/osvschema

1. Add an OSV data fetcher that downloads per-ecosystem OSV data from the osv.dev GCS bucket (https://storage.googleapis.com/osv-vulnerabilities/<ECOSYSTEM>/all.zip) or from distro-specific sources (e.g., Canonical's GitHub repo for Ubuntu).

2. Add a unified OSV parser that reads OSV JSON records and extracts per-CVE affected package information, including binary package names, affected version ranges, and fixed versions per release. Since OSV uses the same schema across all ecosystems, one parser serves all distros.

3. Add an OSV evaluator that matches installed packages (deb or rpm) against OSV records to determine vulnerabilities, replacing the current OVAL evaluation.

• Feature flag to swap between OVAL and OSV
• Kernel packages require special version normalization and matching logic

4. Migrate Ubuntu first: Replace USN OVAL with Ubuntu OSV data. Run in parallel with OVAL during validation to compare results.

OSV data delivery options

1. Full ecosystem download (all.zip)

Each ecosystem has a single zip file containing all vulnerability records:

• https://storage.googleapis.com/osv-vulnerabilities/Ubuntu/all.zip
• https://storage.googleapis.com/osv-vulnerabilities/Debian/all.zip
• https://storage.googleapis.com/osv-vulnerabilities/Red%20Hat/all.zip

Each zip contains individual JSON files, one per CVE record.

2. Incremental updates via modified_id.csv

Each ecosystem directory has a modified_id.csv file sorted in reverse chronological order:

• gs://osv-vulnerabilities/Ubuntu/modified_id.csv
• gs://osv-vulnerabilities/Debian/modified_id.csv

Format: ,<ecosystem_dir>/

You can stream this file and stop as soon as you reach a timestamp you've already processed. Then fetch only the changed individual JSON files at gs://osv-vulnerabilities//.json.

3. REST API (per-query, not bulk)

• POST /v1/query — query by package/version
• POST /v1/querybatch — batch queries
• GET /v1/vulns/{id} — fetch a single record

These are for real-time lookups, not bulk ingestion.

4. https://github.com/canonical/ubuntu-security-notices.git

• git clone --shallow-since="${DAYS_TO_KEEP} days ago"
• use github worker cache to keep repository
• git fetch --update-shallow --shallow-since="${DAYS_TO_KEEP} days ago" origin main
• generate an all & today/yesterday delta artifacts

The most practical approach for Fleet would be:

1. Initial sync: Download all.zip for each ecosystem (~one-time or periodic full refresh)
2. Subsequent syncs: Use modified_id.csv to identify changed records since last sync, then fetch only those individual JSON files — this is lightweight and efficient
3. Store a "last synced" timestamp to know where to stop reading the CSV

Key files to modify/replace:

• server/vulnerabilities/oval/sync.go — Download logic (replace with OSV fetcher)

• server/vulnerabilities/oval/parser.go — Parsing pipeline (replace with OSV parser)

• server/vulnerabilities/oval/parsed/ubuntu_result.go — Ubuntu evaluation (replace)

• server/vulnerabilities/oval/parsed/rhel_result.go — RHEL evaluation (replace)

• server/vulnerabilities/oval/analyzer.go — Scan orchestration (update to use OSV)

• server/vulnerabilities/oval/matching_rules.go — Many rules may become unnecessary

• server/vulnerabilities/oval/input/ — OVAL XML structs (eventually remove)

• Test plan is finalized

• Contributor API changes: No changes

• Feature guide changes: No changes

• Database schema migrations: No changes expected (software_cve table schema remains the same)

• Load testing: Compare OSV scan performance against current OVAL scan across Ubuntu hosts

• Pre-QA load test: Yes

• Load testing/osquery-perf improvements: No changes

• This is a premium only feature: No

QA

Risk assessment

• Risk level: High
• Risk description: This changes the core vulnerability data source for all Ubuntu hosts. Incorrect implementation could cause missed vulnerabilities (false negatives) or new false positives. Each phase requires thorough comparison testing between OVAL and OSV results before switching over.

Primary Risks:

1. False Positives from Kernel Package Bugs (High Impact)

   • Risk: Incorrectly attributing kernel CVEs to linux-headers-*, linux-modules-*, etc.
   • Impact: POC testing showed this can create 227,000+ false positive vulnerabilities
   • Mitigation: Restrict kernel CVE mapping to only linux-image-* and linux-signed-image-*
   • Testing: Unit tests must verify non-image packages don't get kernel CVEs

2. False Negatives from Version Matching (High Impact)

   • Risk: Version format mismatches prevent CVE detection
   • Example: 5.15.0-94-generic (osquery) vs 5.15.0-94.104 (OSV)
   • Impact: Zero kernel CVEs detected if version matching fails
   • Mitigation: Implement kernel version normalization and prefix matching
   • Testing: Integration tests with real kernel packages

3. CISA KEV Coverage Gap (Medium Impact)

   • Risk: OSV misses 67% of CISA Known Exploited Vulnerabilities vs OVAL
   • Impact: POC found OSV detects 14 KEVs vs OVAL's 43 KEVs
   • Mitigation:
     • Document KEV gap in release notes
     • Consider CISA KEV API supplementation
     • Run dual-source validation (OSV + CISA KEV list)

4. Increased Alert Volume (Medium Impact)

   • Risk: OSV finds 27% more CVEs, increasing user alert fatigue
   • Impact: Users may see 20-30% more vulnerability notifications
   • Mitigation:
     • Communicate expected increase in release notes
     • Provide filtering/prioritization features
     • Highlight improved coverage as a benefit

5. Data Source Failures (Low Impact)

   • Risk: GCS bucket unavailable or corrupted downloads
   • Mitigation: Implement retry logic, fallback to cached artifacts
   • Testing: Simulate network failures and verify graceful degradation

Test plan

Phase 1 (Ubuntu)

• Verify CVE-2024-30205 is NOT flagged for emacs-common on Ubuntu 24.04 (the original false positive from Vulnerability false positive: emacs-common #39370)

• Verify CVE-2024-39331 IS still flagged for emacs-common on Ubuntu 24.04 (legitimate vulnerability)

• Compare OSV scan results against current OVAL scan results across multiple Ubuntu versions (20.04, 22.04, 24.04) to identify differences

• Verify that Ubuntu Pro-only fixes are handled correctly

• Verify vulnerability counts don't drop unexpectedly for Ubuntu hosts in dogfood environment

• run osquery perf to load up ubuntu software db

go run ./cmd/osquery-perf/agent.go \
  --enroll_secret <secret> \
  --host_count 15 \
  --os_templates ubuntu_22.04 \
  --software_db_path /<path to fleet>/cmd/osquery-perf/software-library/software.db \
  --common_software_count 5 \
  --unique_software_count 10 \
  --software_query_fail_prob 0.0

• run vulns processing with OVAL

FLEET_VULNERABILITIES_OSV_FOR_VULNERABILITIES=false  ./build/fleet vuln_processing --dev --dev_license

• run vulns processing with OSV

./build/fleet vuln_processing --dev --dev_license

compare results in software_cve table

General

• Test with hosts running multiple distro versions to ensure per-release granularity works
• Verify that the software_cve source column correctly identifies OSV-sourced vulnerabilities
• Load test with large fleet to ensure OSV processing performance is acceptable

Testing notes

• fleet server pulls down all file and processes
• fleet server pulls down delta files and processes

Confirmation

1. Engineer: Added comment to user story confirming successful completion of test plan.
2. QA: Added comment to user story confirming successful completion of test plan.

[noahtalerman]

Hey @getvictor this sounds like a good investment if it will help fix false positives.

Because there are no interface changes, I think this story qualifies as an engineering initiated story: https://fleetdm.com/handbook/engineering#create-an-engineering-initiated-story

Up to @lukeheath on whether we prioritize

[getvictor]

@noahtalerman I do not think this qualifies for engineering-initiated because it is a customer issue. It should have customer-faltona label: https://fleetdm.slack.com/archives/C044WQFPRCH/p1770156361038339