Team policy automation web hook set through UI not overwritten by GitOps

[allenhouchins]

Fleet version: 4.60.1

Web browser and operating system: Any

---

💥  Actual behavior

https://fleetdm.slack.com/archives/C071NNMSP2R/p1733416063457349
I was working to resolve this issue. I noticed the yaml file we have that controls policy automations was configured to not include any policies (policy ids).

webhook_settings:
failing_policies_webhook:
destination_url: $DOGFOOD_FAILING_POLICIES_WEBHOOK_URL
enable_failing_policies_webhook: true
host_batch_size: 0
policy_ids: []

It seems that policy automation that Victor referenced in Slack must have been set through the UI. I would have expected it to return to the state defined in the yaml during the nightly refresh. I have since unchecked the box for the policy automation in the UI to resolve the reported issue.

https://github.com/fleetdm/fleet/blob/3737e5b75b55b7924dd89f5cf565f39f83aca962/it-and-security/default.yml#L63C1-L68C21

🧑‍💻  Steps to reproduce

1. Create a policy and policy automation utilizing a webhook in the UI.
2. Via gitops, configure webhook settings but don't apply them to any policy ids.
3. Apply settings and notice the webhook automation is still configured.

🕯️ More info (optional)

This was the policy in dogfood: https://dogfood.fleetdm.com/policies/3917?team_id=9

[lukeheath]

@allenhouchins Thanks for filing! Note you can follow the fast-track for Fleeties and put bugs directly in the reproduced state.

[iansltx]

So, this is actually team webhook settings, not global webhook settings, so we're looking at this file. Team webhook/integration settings are managed separately from global settings.

With that said, currently while GitOps parses failing policy webhook configuration, it doesn't actually marshall that config into the spec that gets sent to the server to edit the team. Even if it did send it, the server currently doesn't look for the field on the endpoint that GitOps uses (POST /spec/teams), which is distinct from the endpoint the UI uses to edit team automation configuration (PATCH /teams/{id}). The host status webhook is parsed and updated, but only that config.

On top of this, the spec endpoint follows PATCH semantics: if you don't include the host status webhook in the payload, it won't be modified.

I'm guessing the desired behavior here is:

1. Pass the policy failure webhook spec to the team spec endpoint
2. Implement PATCH semantics for this field on the team spec endpoint
3. If team policy failure webhook settings aren't supplied in a team YAML file, treat this as if the admin intends to zero out the webhook config and disable the webhook. This does contitute a change from eisting behavior, where "no config" means "don't change it", but I'm guessing that we want the revised behavior since this is supposed to be declarative configuration.

@lukeheath @allenhouchins does the above sound correct in terms of the desired fix under the hood?

[allenhouchins]

@iansltx Thanks for catching the nuance between team vs global webhook settings and for the detailed analysis. I will always be in favor of treating our gitops/yaml as declarative configuration. It's a key differentiator in the market to be able to point back to the exact configuration at any moment in time -- allowing a customer to revert to a known good state or enabling them to prove configuration for audit.

[lukeheath]

@iansltx Let's route this ticket through the product drafting board to get feedback on desired behavior. I'm assigning @noahtalerman and moving to drafting.