Test impact of turning on host_async_processing for CIS benchmarks

[zhumo]

This issue's remaining effort can be completed in ≤1 sprint. It will be valuable even if nothing else ships.

It is planned and ready to implement. It is on the proper kanban board.

Goal

User story
As a fleet admin,
I want to know that turning on FLEET_OSQUERY_ENABLE_ASYNC_HOST_PROCESSING=policy_membership=true to enable my CIS benchmark policies will not affect the rest of my fleet experience,
so that I can protect my endpoints without affecting the stability of my fleet instance or impacting end user experience.

How to run the test

Run Fleet locally (you should see the below log line [...]"async enabled, ..." name=policy_membership):

./build/fleet serve --dev --logging_debug \
  --dev_license \
  --osquery_enable_async_host_processing "policy_membership=true" 2>&1 | tee ~/fleet.txt
[...]
level=debug ts=2023-07-10T18:36:46.115005Z cron=async_task
task="async enabled, starting collectors" name=policy_membership interval=30s jitter=10

Metrics to capture

All features related to policies (automations, host refetching, policies tab, policies tab in host details) should work as expected (as if the flag was set to off, which is the default)

There shouldn't be any noticeable performance changes server side when the number of hosts is low.

What potential qualitative issues to watch out for

Emphasis in policies automations and host refetch, because of this comment in https://github.com/fleetdm/fleet/blob/main/docs/Deploying/Configuration.md#osquery_enable_async_host_processing:

Note that currently, if both the failing policies webhook and this osquery.enable_async_host_processing option are set, some failing policies webhooks could be missing (some transitions from succeeding to failing or vice-versa could happen without triggering a webhook request).

Changes

This issue's estimation includes completing:

• UI changes: TODO
• CLI usage changes: TODO
• REST API changes: TODO
• Permissions changes: TODO
• Database schema migrations: TODO
• Outdated documentation changes: TODO
• Scope transparency changes? TODO
• Breaking changes requiring major version bump? TODO
• Changes to paid features or tiers? TODO
• QA complete?
• ...

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

Context

• Requestor(s): _________________________

QA

Risk assessment

• Requires load testing TODO

Risk level: Low / High TODO

Risk description: TODO

Automated:

• Fleet: Cover / Will not cover
• QAWolf: Cover / Will not cover

Manual testing steps

1. Step 1
2. Step 2
3. Step 3

Testing notes

Confirmation

1. Engineer (@____): Added comment to user story confirming succesful completion of QA.
2. QA (@____): Added comment to user story confirming succesful completion of QA.

[zhumo]

Hi @zayhanlon, unfortunately, we were not able to get to this work in our 6-week timeframe. Please bring this back to Feature Fest if it's still desired. Thanks!

[xpkoala]

This flag was enabled for over 48 hours and no issues were found with the use of policies or other functionality in the app.

[noahtalerman]

Confirm and celebrate: @zhumo do we want docs for this? Does this PR cover it? https://github.com/fleetdm/fleet/pull/13799/files

[zhumo]

No, i don't think it does. documented here: #13966

[fleet-release]

CIS benchmarks on,
Stability remains strong,
Fleet in harmony.