FA rollout experiment fixes

[levan-m]

What does this PR do?

Addressing several bugs/issues:

• When changing same config back and forth to rollout, two controller revisions basically get version bump reusing same name/hash. This leads to accumulation of both marker annotations if one config update is rolled back.

      operator.datadoghq.com/experiment-promoted: "true"
      operator.datadoghq.com/experiment-rollback: "true"

• Again when flipping same config, revisions of two controllerrevs increase while their creationTimestamp which is used as proxy for experiment start time, doesn't change. Once these revisions are old enough controller will timeout and rollback.
• If experiment times out, subsequent experiments will fail unless different config change to force new controller rev creation.
• Operator doesn't send task abort or timeout errors to FA (regardless how FA processes those, which now are ignored).

Easiest to review Commit by commit

b6ef9e5 Decouples experiment timing from ControllerRevision metadata by adding explicit StartedAt field to ExperimentStatus. rev.CreationTimestamp might be stale if new spec has is same as earlier experiment. This could lead to immediate timeout if spec matched pre-existing controller rev spec.

d202cf7 rehydrate installer state from DatadogAgent on daemon startup to report state with correct task ID instead of empty one forcing FA to send start requests.

6cf24ad report TaskState_ERROR for the original start task on timeout.

8509c37 replaces two annotations (*experiment-promoted, *experiment-rollback) on controller rev with a single annotation to avoid accidentally accumulating two mutually exclusive annotations and to simplify logic. Bug reproducible by flipping same config back and forth and intentionally failing experiment to force rollback.

5aa1b51 similar to 6cf24ad, report abort error for the experiment task when DDA is changed during experiment.

Motivation

What inspired you to submit this pull request?

Additional Notes

Anything else we should know when reviewing?

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

• Agent: vX.Y.Z
• Cluster Agent: vX.Y.Z

Describe your test plan

Below args needs to be set on Operator (debug level logging recommended)

        - --createControllerRevisions=true
        - --remoteUpdatesEnabled=true
        - --remoteConfigEnabled=true

and env vars to configure cluster name and endpoint

        - name: DD_CLUSTER_NAME
          value: my-test-cluster
        - name: DD_SITE
          value: datad0g.com          # endpoint for internal testing and api/app key in the same org

Happy path

1. Apply DDA
2. Go to FA -> find cluster -> Configuration -> Edit -> Deploy
3. Switch to Deployments view in FA see it succeeds.
4. Check the change was applied in cluster.

Failures
timeout
2. Watch dda yaml | yq .status.experiment, once experiment is running scale Operator to 0 replicas.
3. Wait 15min
4. Scale back to 1
5. Operator marks experiment as timed_out and is rolled back. FA retries same experiment eventually failing deployment.
6. try happy path

abort
2. Watch dda yaml | yq .status.experiment, once experiment is running scale Operator to 0 replicas.
3. Apply updated DDA in cluster
4. Operator marks experiment as aborted, updated DDA should be reconciled in cluster. FA retries same experiment eventually failing deployment.
5. Try happy path.

Checklist

• PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
• PR has a milestone or the qa/skip-qa label
• All commits are signed (see: signing commits)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 79.01235% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 42.31%. Comparing base (f70667d) to head (649b95d).

Files with missing lines	Patch %	Lines
pkg/fleet/daemon.go	65.38%	6 Missing and 3 partials ⚠️
internal/controller/datadogagent/experiment.go	76.92%	4 Missing and 2 partials ⚠️
pkg/fleet/daemon_worker.go	92.85%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3035      +/-   ##
==========================================
+ Coverage   42.24%   42.31%   +0.06%     
==========================================
  Files         337      337              
  Lines       28951    29007      +56     
==========================================
+ Hits        12230    12273      +43     
- Misses      15916    15924       +8     
- Partials      805      810       +5     

Flag	Coverage Δ
unittests	42.31% <79.01%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
internal/controller/datadogagent/revision.go	78.46% <100.00%> (ø)
pkg/fleet/daemon_worker.go	78.18% <92.85%> (+1.95%)	⬆️
internal/controller/datadogagent/experiment.go	77.52% <76.92%> (-0.08%)	⬇️
pkg/fleet/daemon.go	69.34% <65.38%> (-0.60%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f70667d...649b95d. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[datadog-datadog-prod-us1-2]

✨ Fix all issues with BitsAI

🛑 Gate Violations

🎯 1 Code Coverage issue detected

A Patch coverage percentage gate may be blocking this PR.

• Patch coverage: 76.71% (threshold: 80.00%)

ℹ️ Info

🎯 Code Coverage (details)
• Patch Coverage: 76.71%
• Overall Coverage: 42.64% (+0.06%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 649b95d | Docs | Datadog PR Page | Give us feedback!}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5aa1b51cb3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[Mathew-Estafanous]

Instead of doing this, can we use cache.WaitForCacheSync to guarantee the informer is populated?

[levan-m]

I think using APIReader for direct reads and doing it before d.rcClient.Subscribe makes intent clearer. Technically we could use cache.WaitForCacheSync and then read DDA from cache and initialize config version, would be same amount of code but RC subscription would block on whole cache sync vs single DDA list. In practice there is no big tradeoff.
Dropping this and relying only on WaitForCacheSync and informer event triggering config update would remove some code but create race between RC callbacks and informer events.