os_versions endpoint performance improvements

[getvictor]

Related issue: Resolves #34500 and Resolves #33758

Video demo: https://www.youtube.com/watch?v=4HZlKG0G1B0

• Added a new aggregation table operating_system_version_vulnerabilities for faster queries. The table is currently used only for Linux vulnerabilities, but could be used for other OS vulnerabilities.
• Added max_vulnerabilities parameter per API doc
• Also added max_vulnerabilities parameter to os_versions/{id} endpoint, but not making it public since that endpoint is still slow and needs other API changes. bug Missing docs: max_vulnerabilities in GET os_versions/:id and GET /os_versions #34974
• Removed "kernels": [] from os_versions endpoint result

Checklist for submitter

If some of the following don't apply, delete the relevant line.

• Changes file added for user-visible changes in changes/, orbit/changes/ or ee/fleetd-chrome/changes.

• Input data is properly validated, SELECT * is avoided, SQL injection is prevented (using placeholders for values in statements)

Testing

• Added/updated automated tests

• Where appropriate, automated tests simulate multiple hosts and test for host isolation (updates to one hosts's records do not affect another)

• QA'd all new/changed functionality manually

Database migrations

• Checked schema for all modified table for columns that will auto-update timestamps during migration.
• Confirmed that updating the timestamps is acceptable, and will not cause unwanted side effects.
• Ensured the correct collation is explicitly set for character columns (COLLATE utf8mb4_unicode_ci).

Summary by CodeRabbit

• New Features

  • Added ability to limit the number of vulnerabilities displayed for operating system versions via an optional parameter.
  • Introduced vulnerability count tracking for operating system versions, now visible in API responses and UI displays.
  • Enhanced operating system vulnerability visualization with improved count-based rendering.

• Tests

  • Added comprehensive test coverage for vulnerability limiting behavior across multiple operating system versions and architectures.

[mostlikelee]

Great choice on the caching approach. My initial reaction, though, is that we’re trading a fair bit of complexity for performance gains, and wondering if there are ways to simplify.

It might make sense to further decouple Linux vulnerabilities from the rest of the query path since they behave so differently.

[mostlikelee]

we're not deduping here, guessing that will result in over counting?

[getvictor]

operating_system_vulnerabilities should only have non-Linux vulnerabilities and operating_system_version_vulnerabilities should only have Linux vulnerabilities.

operating_system_vulnerabilities are being deduped with GROUP BY v.cve. operating_system_version_vulnerabilities are deduped during aggregation, so they shouldn't even need GROUP BY. But I wasn't focusing on optimizing this os_versions/{id} endpoint, so I think it is OK to leave as is and re-review in the follow up bug.

[mostlikelee]

glad you broke this out, much easier to reason about

[getvictor]

Great choice on the caching approach. My initial reaction, though, is that we’re trading a fair bit of complexity for performance gains, and wondering if there are ways to simplify.

It might make sense to further decouple Linux vulnerabilities from the rest of the query path since they behave so differently.

@mostlikelee Yes, it is complex. We could eliminate operating_system_vulnerabilities and just use operating_system_version_vulnerabilities, or we could decouple by making Linix and non-Linux separate Datastore or Service layer methods.

[mostlikelee]

@mostlikelee Yes, it is complex. We could eliminate operating_system_vulnerabilities and just use operating_system_version_vulnerabilities, or we could decouple by making Linix and non-Linux separate Datastore or Service layer methods.

@getvictor Up to you, as I know this is an urgent request. We could look into refactoring down the line.