Cleanup stale Agents not matching DAP when DDAI is enabled

[levan-m]

What does this PR do?

Adds logic to DDAI reconciler to cleanup Agents which don't match pod affinity of DAP they were created for (due to label changes). DDA path had this logic and wasn't carried over due to this unresolved bug kubernetes/kubernetes#129196. This change primarily targets k8s distros where kubernetes/kubernetes#129196 is resolved.

CONTP-1338

Motivation

What inspired you to submit this pull request?

Additional Notes

Anything else we should know when reviewing?

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

• Agent: vX.Y.Z
• Cluster Agent: vX.Y.Z

Describe your test plan

Test below scenario on a distro where kubernetes/kubernetes#129196 is not fixed, first with Operator 1.22-1.23, then fixed version.

1. Create DAP

apiVersion: datadoghq.com/v1alpha1
kind: DatadogAgentProfile
metadata:
  name: stale-agent-test
  namespace: datadog-agent
spec:
  config:
    override:
      nodeAgent:
        containers:
          agent:
            env:
            - name: STALE_AGENT_TEST
              value: "true"
  profileAffinity:
    profileNodeAffinity:
    - key: agent-profile.datadoghq.com/name
      operator: In
      values:
      - stale-agent-test

2. label nodes

k label node nodeA agent-profile.datadoghq.com/name=stale-agent-test
k label node nodeB agent-profile.datadoghq.com/name=stale-agent-test

3. Watch pods:

• With Operator 1.22, 1.23 new DS will be created with DAP affinity, creating two pods which will be stuck in pending.
• With fix, two pods will be terminated from default DS and ones created from DAP will reach running.

Checklist

• PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
• PR has a milestone or the qa/skip-qa label
• All commits are signed (see: signing commits)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 72.22222% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 38.83%. Comparing base (355ade2) to head (eff9e19).

Files with missing lines	Patch %	Lines
...troller/datadogagent/controller_reconcile_agent.go	76.47%	4 Missing and 4 partials ⚠️
internal/controller/datadogagent/profile.go	0.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2657      +/-   ##
==========================================
+ Coverage   38.69%   38.83%   +0.13%     
==========================================
  Files         307      307              
  Lines       26563    26599      +36     
==========================================
+ Hits        10278    10329      +51     
+ Misses      15511    15490      -21     
- Partials      774      780       +6     

Flag	Coverage Δ
unittests	38.83% <72.22%> (+0.13%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
internal/controller/datadogagent/profile.go	72.60% <0.00%> (-1.01%)	⬇️
...troller/datadogagent/controller_reconcile_agent.go	66.45% <76.47%> (+4.04%)	⬆️

... and 2 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 355ade2...eff9e19. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.