Vuln processing command

[edwardsb]

Goal

Today Fleet runs vulnerability downloading and processing via internal scheduled cron job. This internal mechanism is very useful for frictionless deployments and is well suited for most use cases. However I think it would be helpful to have the ability to externally schedule this command, in similar fashion to fleet prepare db, where the command runs to completion and exists gracefully.

It is possible to use https://fleetdm.com/docs/deploying/configuration#current-instance-checks to limit specific instances to process vuln data, but this still requires the instance deployed with this setting to run 24/7. It could be possible to schedule vuln processing tasks to run via https://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduling_tasks.html in ECS or https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/ in Kubernetes and https://cloud.google.com/run/docs/triggering/using-scheduler#create_job with Google Cloud Run.

What does this accomplish? What is the point?

Vuln processing requires significantly more memory that the fleet web server does. Its recommended at 4GB of Memory. Without limiting which instance of Fleet is elected the leader for a particular round of vuln processing, you are required to run 4GB of memory across your entire deployment, since leader election would be random and any instance of fleet could attempt vuln processing. If you do use the limiting feature, you are still required to run at least one provisioned instanced with 4GB of memory 24/7, even though vuln processing only occurs by default 1h.

Having a dedicated command to processing vulnerabilities enables large deployments to significantly reduce the required memory for the majority of their instances saving on memory and thus costs.

How?

• Introduce a dedicated vulnerability processing command similar to fleet prepare db

Concerns

• Vuln db would need to be downloaded from each time since the command would be starting from scratch in a containerized environment. It would be possible to mitigate this with stateful-sets(?) and https://kubernetes.io/docs/concepts/storage/persistent-volumes/ allowing the vuln processing task to mount a persistent volume, effectively caching the vuln db. This is also achievable in ECS https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_data_volumes.html
• If you are already running dozens of fleet containers, and leader election is seemingly random, its likely that today the vuln db is downloaded a number of times before it starts getting cached anyways.

[edwardsb]

@zwass here is the issue related to vuln processing we discussed.

[zwass]

@edwardsb thank you for filing this.

@noahtalerman @chiiph and I were just discussing all of the different "intervals" in Fleet and I brought up your suggestion about this. We think it probably makes sense to extend this idea to all of the background jobs as part of an effort to create a generic framework for background jobs in Fleet.

[mikermcneil]

it probably makes sense to extend this idea to all of the background jobs as part of an effort to create a generic framework for background jobs in Fleet.

👍

Would be good for it to be opt-in though. So that everything still just works™ unless you need more customization.

cc @CptOfEvilMinions