Vulnerability Processing Taking Longer than expected

[mostlikelee]

Fleet version: 4.44.0

Web browser and operating system: NA

---

💥  Actual behavior

Vulnerability processing was observed taking 2x longer after upgrade to v4.44.0 in some Fleet Cloud instances.

🧑‍💻  Steps to reproduce

Slack thread reference: https://fleetdm.slack.com/archives/C019WG4GH0A/p1707682626450799

This was observed with CPU utilization and spotchecked observing created_at and updated_at timestamps for vulnerability cron jobs in the database.

🕯️ More info (optional)

An increase in scan time is expected when upgrading to 4.44.0 due to the addition of OS vulnerability scans, but 2x seems excessive. As observed in one cloud hosted environment, the total OS count was ~40, and primarily macOS devices.

QA Testplan

1. Run vuln processing before this change and note the vulns identified.
2. Run vuln processing after this change and compare to the above.

[rfairburn]

Logs:

CloudWatch Logs Insights
region: us-east-2
log-group-names: REDACTED
start-time: -21600s
end-time: 0s
query-string:

fields @timestamp, @message, @logStream, @log
| sort @timestamp desc
| filter cron like /vulnerabilities/
| filter msg not like /might still be running/
| filter msg not like /error translating to CPE/
| limit 5

---

@timestamp	@message	@logstream	@log
2024-02-14 21:39:58.087	{"cron":"vulnerabilities","instanceID":"kGeGzhC5+fL4M6y1vEhkW05A4wTYi8Fhw0az3hXLpnT9G//NcPRIsO+3yzIqGGJPjVCH17ebvsT5Z8ncSeRQ+A==","level":"info","msg":"total runtime (2h39m41.05537752s) exceeded schedule interval (1h0m0s)","schedule":"vulnerabilities","ts":"2024-02-14T21:39:58.086985161Z"}	fleet/fleet/11addbc9bdc84b3483ed075a8f2a5f17	611884880216:pinterest
2024-02-14 21:39:58.086	{"cron":"vulnerabilities","instanceID":"kGeGzhC5+fL4M6y1vEhkW05A4wTYi8Fhw0az3hXLpnT9G//NcPRIsO+3yzIqGGJPjVCH17ebvsT5Z8ncSeRQ+A==","level":"info","schedule":"vulnerabilities","status":"completed","ts":"2024-02-14T21:39:58.086847177Z"}	fleet/fleet/11addbc9bdc84b3483ed075a8f2a5f17	611884880216:pinterest
2024-02-14 19:00:17.038	{"cron":"vulnerabilities","instanceID":"kGeGzhC5+fL4M6y1vEhkW05A4wTYi8Fhw0az3hXLpnT9G//NcPRIsO+3yzIqGGJPjVCH17ebvsT5Z8ncSeRQ+A==","level":"info","schedule":"vulnerabilities","status":"pending","ts":"2024-02-14T19:00:17.037640489Z"}	fleet/fleet/11addbc9bdc84b3483ed075a8f2a5f17	611884880216:pinterest
2024-02-14 19:00:17.038	{"cron":"vulnerabilities","level":"info","periodicity":"1h0m0s","ts":"2024-02-14T19:00:17.037686982Z"}	fleet/fleet/11addbc9bdc84b3483ed075a8f2a5f17	611884880216:pinterest
2024-02-14 19:00:17.038	{"cron":"vulnerabilities","level":"info","msg":"scanning vulnerabilities","ts":"2024-02-14T19:00:17.037712783Z"}	fleet/fleet/11addbc9bdc84b3483ed075a8f2a5f17	611884880216:pinterest

---