Add Datadog Agent Profiles

[davidor]

What does this PR do?

Adds support for Datadog Agent Profiles.

It adds a new CRD DatadogAgentProfile that allows users to override the agent resources for the nodes of the cluster that match the given affinity.

This is an example:

apiVersion: datadoghq.com/v1alpha1
kind: DatadogAgentProfile
metadata:
  name: profile-example
spec:
  profileAffinity:
    profileNodeAffinity:
      - key: disktype
        operator: In
        values:
          - ssd
  config:
    override:
      nodeAgent:
        containers:
          agent:
            resources:
              requests:
                cpu: 20m

Describe your test plan

This has already been tested, you can check our internal doc with the test plan for more details.
In general, try the feature and verify that everything works as you would expect.

Checklist

• PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
• PR has a milestone or the qa/skip-qa label

[github-actions]

This pull request does not contain a valid label. Please add one of the following labels: bug, enhancement, refactoring, documentation, tooling, dependencies

[codecov-commenter]

Codecov Report

Attention: 201 lines in your changes are missing coverage. Please review.

Comparison is base (cb2d3ba) 58.54% compared to head (791ae60) 58.74%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1083      +/-   ##
==========================================
+ Coverage   58.54%   58.74%   +0.19%     
==========================================
  Files         166      170       +4     
  Lines       20525    21033     +508     
==========================================
+ Hits        12017    12356     +339     
- Misses       7782     7920     +138     
- Partials      726      757      +31     

Flag	Coverage Δ
unittests	58.74% <65.81%> (+0.19%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...is/datadoghq/v1alpha1/datadogagentprofile_types.go	100.00% <100.00%> (ø)
...tadoghq/v1alpha1/datadogagentprofile_validation.go	100.00% <100.00%> (ø)
...ontrollers/datadogagent/feature/oomkill/feature.go	87.50% <ø> (ø)
controllers/datadogagent/override/container.go	94.70% <100.00%> (+0.26%)	⬆️
controllers/setup.go	60.00% <100.00%> (+2.71%)	⬆️
pkg/kubernetes/const.go	86.95% <ø> (ø)
pkg/kubernetes/provider.go	85.71% <100.00%> (-0.24%)	⬇️
controllers/datadogagent/override/daemonset.go	0.00% <0.00%> (ø)
pkg/kubernetes/objects.go	0.00% <0.00%> (ø)
controllers/datadogagentprofile_controller.go	64.70% <64.70%> (ø)
... and 5 more

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cb2d3ba...791ae60. Read the comment docs.

[levan-m]

I'm a bit confused by error handling here as some errors are collected into errs, some lead to return from function.
In this case, why are we returning here (but not on line 156)?
What happens if some profile/provider combinations succeed in line 162, shouldn't we doe Create/Update/Cleanup dependencies?

[davidor]

I don't think it makes a bit difference in the end because any error here means that the request needs to be requeued and retried. I think it's a bit clearer to ensure that all the agent daemonsets are created as expected and unnecessary ones deleted before trying to manage their dependencies.

For the providers part, @khewonc can answer, I'm not so familiar with that feature.

[khewonc]

When adding introspection, I was also thinking we shouldn't block the rest of the reconcile even if there was an error with handleProviders since an error there would come from an issue with deleting old DaemonSets. Then, when merging main into the profiles branch, I tried to make minimal changes so the different approaches to error handling were kept. We can make the behavior for both consistent in another PR

[levan-m]

This function does node labeling, after DSs are already create. What happens if DS creation is successful but labeling fails (or keeps failing)? Do we reconcile such nodes?

[davidor]

The error handling logic follows the same pattern as in other places of the code.

All the steps in handleProfiles should be successful. If there's an error in any of the steps, we return an error in the Reconcile function so that the request is retried later.

If it keeps failing, we would have the same problem as in any other part of the code that creates, updates or deletes any Kubernetes objects. There isn't much we can do. Although the only case where I think the node labeling could be failing permanently (and not other operations) is if RBACs are not properly set, but that should be taken care of by our installation charts.