[recurring] Update `agent_options` validation for `config` and `team` YAML to match the latest osquery version

[mna]

Goal

Introduced in #5222, Fleet now validates all keys under the agent_options key in the config and team YAML files.

Validation is based on the latest osquery version at the team of the Fleet release.

This means that before each Fleet release, we should update validation if there was a new osquery version released.

This ticket should thus be a recurring task.

How?

Backend

1. Check if a new version of osquery has been released and compare it to the osquery version used for validation in Fleet.

• If there isn't a new osquery release, this ticket is done for this iteration.

2. Autogenerate new structs for validating the Fleet settings under the agent_options.config.options key (in Fleet's config YAML) and the command-line flags under the agent_options.command_line_flags key.

• Use the tools/osquery-agent-options/main.go Go program to automatically generate the new structs corresponding to the new osquery settings and command-line flags. Compare the generated structs to the current structs in the code (the easiest way is probably to replace the structs in the code, and check with git the diff - the structs in question are osqueryOptions and osqueryCommandLineFlags and they are here).
• The Go program is documented here. Note that you have to install the latest osquery before running the tool.
• Note that osquery calls these settings CLI-flags. Only the flags valid on your OS will be generated automatically - there's a separate step to verify the OS-specific flags.

3. Manually update validation for all settings not included under the agent_options.config.options key.

• This includes settings under the following keys:
  • agent_options.file_paths, agent_options.yara, agent_options.prometheus, agent_options.view, agent_options.ec2, agent_options.decorators, agent_options.automatic_table_construction, agent_options.events.
• Check if of these settings changed in the new osquery release by diffing this osquery file between the osquery releases (e.g. on a local clone of the osquery repo, switch to the latest version - for example git checkout 5.5.1 - and run a diff against the previous version on that file, e.g. git difftool 5.4.0 -- ./docs/wiki/deployment/configuration.md).
• If they did, update the osqueryAgentOptions struct accordingly here, and any additional validation that may be required in validateJSONAgentOptionsSet here.
• Note that the agent_options.schedule and agent_options.packs keys should be accepted (can be present) but their value not validated (that is, they can contain anything).

4. Manually check if OS-specific flags have changed for the new osquery version.

• Those flags are usually listed on osquery's "Command-line flags" page, under Linux-only, Windows-only and macOS-only headers: https://osquery.readthedocs.io/en/stable/installation/cli-flags/
• A quick way to see if those have changed is to diff this osquery file between the osquery releases (e.g. on a local clone of the osquery repo, switch to the latest version - for example git checkout 5.5.1 - and run a diff against the previous version on that file, e.g. git difftool 5.4.0 -- ./docs/wiki/installation/cli-flags.md).
• If the flags did change, update the OsqueryCommandLineFlags{Linux,MacOS,Windows} structs accordingly in this file.
• Not all OS-specific flags can be found via the osquery documentation page, so make sure to go through the osquery-agent-options tool README file for up-to-date information on how to catch all those flags: https://github.com/fleetdm/fleet/tree/main/tools/osquery-agent-options

5. If 2), 3) or 4) find any change, then the validations must be modified accordingly, and tests may need to be adjusted. Also, the default agent options we set for new installs may need to be adjusted (see https://github.com/fleetdm/fleet/blob/main/server/fleet/app.go#L261)

Frontend

Generally there should be no impact on the frontend (some smoke tests should probably be done to ensure we don't show invalid options in help messages or as part of default yaml or json options).

Notes

It's hard to define how often we'll have to go through all steps - some osquery releases have come very quickly (5.1->5.2 within a month), while others have taken many months (5.2->5.3 was ~5 months).

Osquery does not drop config options too often. New options are added relatively often. So risk of Fleet config validation preventing the user from applying changes to past osquery versions is low. Users also have the --force option in case of some issue here.

[mna]

@noahtalerman @lukeheath I created this to track ongoing work required to keep #5222 validations up to date, feel free to close if you prefer a different tracking solution.

[zhumo]

Meeting w/ Noah:

1. Osquery does not drop config options too often, so risk of mismatch between config and osquery is low. Users also have the --force option in case of some issue here.

[noahtalerman]

2. Check for any changes in the configuration settings
3. For CLI flags that can be set as configuration options in the Agent Options, use the tools/osquery-agent-options/main.go Go program to automatically generate the new struct corresponding to those flags

@mna if step 3 automatically generates the new struct, do we need step 2? I might be missing something.

[noahtalerman]

cc @zhumo ^^

[mna]

@noahtalerman Yes, that's because the automatically-generated part covered by 3) is just for CLI flags that can be used in the "options" key of the Agent Options (it's the "options" section documented here: https://osquery.readthedocs.io/en/stable/deployment/configuration/#options), but the rest of the validations need to be manually verified in the rest of that documentation page, e.g. the "schedule" section and subsequent sections: https://osquery.readthedocs.io/en/stable/deployment/configuration/#schedule).