gpu: fix operator deployment instructions #20552

gjulianm · 2025-06-19T11:39:14Z

What does this PR do?

Updates the GPU deployment instructions to fix the deployment with the Datadog operator.

Motivation

Review checklist (to be filled by reviewers)

Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

janine-c

Looks great! I made some minor writing suggestions to try to make the directions a bit easier to parse and to follow our style guide a little more closely. If you have any questions, don't hesitate to let me know!

janine-c · 2025-06-19T17:16:22Z

gpu/README.md

 ```

-For **mixed environments**, use the [DatadogAgentProfiles feature](https://github.com/DataDog/datadog-operator/blob/main/docs/datadog_agent_profiles.md) of the operator, which allows different configurations to be deployed for different nodes. In this case, it is not necessary to modify the DatadogAgent manifest. Instead, create a profile that enables the configuration on GPU nodes only:
+For **mixed environments**, use the [DatadogAgentProfiles (DAP) feature](https://github.com/DataDog/datadog-operator/blob/main/docs/datadog_agent_profiles.md) of the operator, which allows different configurations to be deployed for different nodes. Note that this feature is disabled by default, so it needs to be enabled [as described here](https://github.com/DataDog/datadog-operator/blob/main/docs/datadog_agent_profiles.md#enabling-datadogagentprofiles).


Suggested change

For **mixed environments**, use the [DatadogAgentProfiles (DAP) feature](https://github.com/DataDog/datadog-operator/blob/main/docs/datadog_agent_profiles.md) of the operator, which allows different configurations to be deployed for different nodes. Note that this feature is disabled by default, so it needs to be enabled [as described here](https://github.com/DataDog/datadog-operator/blob/main/docs/datadog_agent_profiles.md#enabling-datadogagentprofiles).

For **mixed environments**, use the [DatadogAgentProfiles (DAP) feature](https://github.com/DataDog/datadog-operator/blob/main/docs/datadog_agent_profiles.md) of the operator, which allows different configurations to be deployed for different nodes. Note that this feature is disabled by default, so it needs to be enabled. For more information, see [Enabling DatadogAgentProfiles](https://github.com/DataDog/datadog-operator/blob/main/docs/datadog_agent_profiles.md#enabling-datadogagentprofiles).

Making the link more descriptive, which is a good accessibility practice 🙂

janine-c · 2025-06-19T17:42:49Z

gpu/README.md

+Modifying the DatadogAgent manifest is necessary to enable certain features that are not supported by the DAP yet. First, the existing configuration should enable the `system-probe` container in the datadog-agent pods (this can be easily checked by looking at the list of containers when running `kubectl describe pod <datadog-agent-pod-name> -n <namespace>`). Because the DAP feature does not yet support conditionally enabling containers, a feature that uses `system-probe` needs to be enabled for all agent pods. We recommend enabling the `oomKill` integration, as it is lightweight and does not require any additional configuration or extra cost.
+
+Additionally, the agent needs to be configured so that the NVIDIA container runtime exposes GPUs to the agent. This can be done via environment variables or volume mounts, depending on whether the `accept-nvidia-visible-devices-as-volume-mounts` parameter is set to `true` or `false` in the NVIDIA container runtime configuration. We recommend configuring the agent both ways, as it reduces the chance of misconfiguration and there are no side effects to having both.
+
+Also, the PodResources socket needs to be exposed to the agent too to integrate with the Kubernetes Device Plugin. Again, this needs to be done globally as the DAP does not yet support conditional volume mounts.


Suggested change

Modifying the DatadogAgent manifest is necessary to enable certain features that are not supported by the DAP yet. First, the existing configuration should enable the `system-probe` container in the datadog-agent pods (this can be easily checked by looking at the list of containers when running `kubectl describe pod <datadog-agent-pod-name> -n <namespace>`). Because the DAP feature does not yet support conditionally enabling containers, a feature that uses `system-probe` needs to be enabled for all agent pods. We recommend enabling the `oomKill` integration, as it is lightweight and does not require any additional configuration or extra cost.

Additionally, the agent needs to be configured so that the NVIDIA container runtime exposes GPUs to the agent. This can be done via environment variables or volume mounts, depending on whether the `accept-nvidia-visible-devices-as-volume-mounts` parameter is set to `true` or `false` in the NVIDIA container runtime configuration. We recommend configuring the agent both ways, as it reduces the chance of misconfiguration and there are no side effects to having both.

Also, the PodResources socket needs to be exposed to the agent too to integrate with the Kubernetes Device Plugin. Again, this needs to be done globally as the DAP does not yet support conditional volume mounts.

Modifying the DatadogAgent manifest is necessary to enable certain features that are not supported by the DAP yet:

- In the existing configuration, enable the `system-probe` container in the datadog-agent pods. Because the DAP feature does not yet support conditionally enabling containers, a feature that uses `system-probe` needs to be enabled for all Agent pods.

- You can check this by looking at the list of containers when running `kubectl describe pod <datadog-agent-pod-name> -n <namespace>`.

- Datadog recommends enabling the `oomKill` integration, as it is lightweight and does not require any additional configuration or cost.

- Configure the Agent so that the NVIDIA container runtime exposes GPUs to the Agent.

- You can do this using environment variables or volume mounts, depending on whether the `accept-nvidia-visible-devices-as-volume-mounts` parameter is set to `true` or `false` in the NVIDIA container runtime configuration.

- Datadog recommends configuring the Agent both ways, as it reduces the chance of misconfiguration. There are no side effects to having both.

- Expose the PodResources socket to the Agent to integrate with the Kubernetes Device Plugin.

- This needs to be done globally, as the DAP does not yet support conditional volume mounts.

I tried formatting this section a little bit differently, in the hopes of making it easier for customers to understand the action items they need to do.

agree with this one, short bullet one liners are easier to follow and not miss, rather than long paragraphs of plain text.

janine-c · 2025-06-19T17:43:03Z

gpu/README.md

+```yaml
+spec:
+  features:
+    oomKill: # Only enable this feature if there is nothing else that requires the system-probe container in all agent pods


Suggested change

oomKill: # Only enable this feature if there is nothing else that requires the system-probe container in all agent pods

oomKill: # Only enable this feature if there is nothing else that requires the system-probe container in all Agent pods

why do we need to enable oomkill for SP? enabling GPUM is not enough?

val06

reviewed

val06 · 2025-06-20T08:20:29Z

gpu/README.md

+```yaml
+spec:
+  features:
+    oomKill: # Only enable this feature if there is nothing else that requires the system-probe container in all agent pods


why do we need to enable oomkill for SP? enabling GPUM is not enough?

val06 · 2025-06-20T08:23:49Z

gpu/README.md

+              readOnly: true
+            - name: pod-resources
+              mountPath: /var/lib/kubelet/pod-resources
+              readOnly: true


i vaguely remember that we need it as rw (ref)

val06 · 2025-06-20T08:24:41Z

gpu/README.md

          agent:
            env:
              - name: DD_ENABLE_NVML_DETECTION
-                value: "true" 
-              # add this env var, if using operator versions 1.14.x or 1.15.x
+                value: "true"


why removing the original comment?
this env is required for customers that still use operator version 1.14, as you added it explicitly to the gpu/feature only in 1.15

val06 · 2025-06-20T08:24:59Z

gpu/README.md

          agent:
            env:
              - name: DD_ENABLE_NVML_DETECTION
-                value: "true" 
-              # add this env var, if using operator versions 1.14.x or 1.15.x


Update operator instructions

0c94647

gjulianm self-assigned this Jun 19, 2025

temporal-github-worker-1 bot added docs/review-requested ecosystems/review-requested product/review-requested labels Jun 19, 2025

datadog-agent-integrations-bot bot added documentation integration/gpu labels Jun 19, 2025

gjulianm marked this pull request as ready for review June 19, 2025 11:39

gjulianm requested review from a team as code owners June 19, 2025 11:39

datadog-agent-integrations-bot bot added team/documentation team/ebpf-platform labels Jun 19, 2025

gjulianm added the qa/skip-qa Automatically skip this PR for the next QA label Jun 19, 2025

janine-c approved these changes Jun 19, 2025

View reviewed changes

temporal-github-worker-1 bot added docs/approved and removed docs/review-requested labels Jun 19, 2025

val06 requested changes Jun 20, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

gpu: fix operator deployment instructions #20552

gpu: fix operator deployment instructions #20552

Uh oh!

gjulianm commented Jun 19, 2025 •

edited

Loading

Uh oh!

janine-c left a comment

Uh oh!

janine-c Jun 19, 2025

Uh oh!

janine-c Jun 19, 2025

Uh oh!

val06 Jun 20, 2025

Uh oh!

janine-c Jun 19, 2025

Uh oh!

val06 Jun 20, 2025

Uh oh!

val06 left a comment

Uh oh!

val06 Jun 20, 2025

Uh oh!

val06 Jun 20, 2025

Uh oh!

val06 Jun 20, 2025

Uh oh!

val06 Jun 20, 2025

Uh oh!

Uh oh!

-Modifying the DatadogAgent manifest is necessary to enable certain features that are not supported by the DAP yet. First, the existing configuration should enable the `system-probe` container in the datadog-agent pods (this can be easily checked by looking at the list of containers when running `kubectl describe pod <datadog-agent-pod-name> -n <namespace>`). Because the DAP feature does not yet support conditionally enabling containers, a feature that uses `system-probe` needs to be enabled for all agent pods. We recommend enabling the `oomKill` integration, as it is lightweight and does not require any additional configuration or extra cost.
-Additionally, the agent needs to be configured so that the NVIDIA container runtime exposes GPUs to the agent. This can be done via environment variables or volume mounts, depending on whether the `accept-nvidia-visible-devices-as-volume-mounts` parameter is set to `true` or `false` in the NVIDIA container runtime configuration. We recommend configuring the agent both ways, as it reduces the chance of misconfiguration and there are no side effects to having both.
-Also, the PodResources socket needs to be exposed to the agent too to integrate with the Kubernetes Device Plugin. Again, this needs to be done globally as the DAP does not yet support conditional volume mounts.
+Modifying the DatadogAgent manifest is necessary to enable certain features that are not supported by the DAP yet:
+- In the existing configuration, enable the `system-probe` container in the datadog-agent pods. Because the DAP feature does not yet support conditionally enabling containers, a feature that uses `system-probe` needs to be enabled for all Agent pods.
+  - You can check this by looking at the list of containers when running `kubectl describe pod <datadog-agent-pod-name> -n <namespace>`.
+  - Datadog recommends enabling the `oomKill` integration, as it is lightweight and does not require any additional configuration or cost.
+- Configure the Agent so that the NVIDIA container runtime exposes GPUs to the Agent.
+  - You can do this using environment variables or volume mounts, depending on whether the `accept-nvidia-visible-devices-as-volume-mounts` parameter is set to `true` or `false` in the NVIDIA container runtime configuration.
+  - Datadog recommends configuring the Agent both ways, as it reduces the chance of misconfiguration. There are no side effects to having both.
+- Expose the PodResources socket to the Agent to integrate with the Kubernetes Device Plugin.
+  - This needs to be done globally, as the DAP does not yet support conditional volume mounts.

	oomKill: # Only enable this feature if there is nothing else that requires the system-probe container in all agent pods
	oomKill: # Only enable this feature if there is nothing else that requires the system-probe container in all Agent pods

gpu: fix operator deployment instructions #20552

Are you sure you want to change the base?

gpu: fix operator deployment instructions #20552

Uh oh!

Conversation

gjulianm commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Review checklist (to be filled by reviewers)

Uh oh!

janine-c left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

val06 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gjulianm commented Jun 19, 2025 •

edited

Loading