Add documentation for the systemd nvidia-container-toolkit.service #203

ArangoGutierrez · 2025-06-27T11:18:18Z

No description provided.

github-actions · 2025-06-27T11:21:32Z

Documentation preview

https://nvidia.github.io/cloud-native-docs/review/pr-203

elezar · 2025-06-27T11:28:00Z

container-toolkit/cdi-support.md

@@ -29,7 +30,68 @@ CDI also improves the compatibility of the NVIDIA container stack with certain f

 - You installed an NVIDIA GPU Driver.

-### Procedure
+### Automatic CDI Specification Generation (v1.18.0+)


Suggested change

### Automatic CDI Specification Generation (v1.18.0+)

### Automatic CDI Specification Generation

elezar · 2025-06-27T11:28:46Z

container-toolkit/cdi-support.md

+As of NVIDIA Container Toolkit `v1.18.0`, the CDI specification is automatically generated and updated by a systemd service called `nvidia-cdi-refresh`. This service:
+
+- Automatically generates the CDI specification at `/var/run/cdi/nvidia.yaml` when NVIDIA drivers are installed or upgraded
+- Monitors changes to driver-related files (`modules.dep` and `modules.dep.bin`) to trigger regeneration


I think this can be removed.

elezar · 2025-06-27T11:29:03Z

container-toolkit/cdi-support.md

+- Automatically generates the CDI specification at `/var/run/cdi/nvidia.yaml` when NVIDIA drivers are installed or upgraded
+- Monitors changes to driver-related files (`modules.dep` and `modules.dep.bin`) to trigger regeneration
+- Runs automatically on system boot to ensure the specification is up to date
+- Is enabled and started automatically during fresh installations


Suggested change

- Is enabled and started automatically during fresh installations

It should be enabled at all times.

elezar · 2025-06-27T11:29:24Z

container-toolkit/cdi-support.md

+The automatic service handles the following scenarios:
+- First-time driver installation
+- Driver upgrades
+- System reboots


Is this not just saying the same as the points above?

elezar · 2025-06-27T11:29:58Z

container-toolkit/cdi-support.md

+- Runtime topology changes (MIG, hot-plug, module unload/load)
+- Manual configuration changes
+
+For these scenarios, you may still need to manually regenerate the CDI specification.


Can we link to the instructions here?

elezar · 2025-06-27T11:30:15Z

container-toolkit/cdi-support.md

+The automatic CDI refresh service does not handle:
+- Driver removal (the CDI file is intentionally preserved)
+- Runtime topology changes (MIG, hot-plug, module unload/load)
+- Manual configuration changes


What are "Manual configuration changes"?

elezar · 2025-06-27T11:30:55Z

container-toolkit/cdi-support.md

+```{note}
+The automatic CDI refresh service does not handle:
+- Driver removal (the CDI file is intentionally preserved)
+- Runtime topology changes (MIG, hot-plug, module unload/load)


Let's just replace this with "MIG device reconfiguration".

elezar · 2025-06-27T11:31:40Z

container-toolkit/cdi-support.md

+```bash
+# /etc/nvidia-container-toolkit/cdi-refresh.env
+NVIDIA_CTK_DEBUG=1
+# Add other nvidia-ctk environment variables as needed


Let's reference the nvidia-ctk cdi generate command to give a list of envvars.

Is a systemctl daemon-reload needed to reload the env?

elezar · 2025-06-27T11:32:11Z

container-toolkit/cdi-support.md

+# Check service status
+$ sudo systemctl status nvidia-cdi-refresh.path
+$ sudo systemctl status nvidia-cdi-refresh.service


Let's add example output here

elezar · 2025-06-27T11:33:10Z

container-toolkit/cdi-support.md

+# View service logs
+$ sudo journalctl -u nvidia-cdi-refresh.service
+```


Split these into separate sections.

+1

We typically split up the sample command and the example output so its clearer what folks need to run

elezar · 2025-06-27T11:34:07Z

container-toolkit/cdi-support.md

+# Manually trigger CDI generation
+$ sudo systemctl start nvidia-cdi-refresh.service


Question: Why are we listing this here and below in the manual generation section?

yeah, on a second thought I decided to remove this

elezar · 2025-06-27T11:34:42Z

container-toolkit/cdi-support.md

+# Manually trigger CDI generation
+$ sudo systemctl start nvidia-cdi-refresh.service
+
+# Enable/disable the automatic monitoring


Let's put this in a separate code block.

Also: What does "automatic monitoring" mean?

elezar · 2025-06-27T11:34:53Z

container-toolkit/cdi-support.md

+$ sudo systemctl start nvidia-cdi-refresh.service
+
+# Enable/disable the automatic monitoring
+$ sudo systemctl enable nvidia-cdi-refresh.path


Should we add the --now flag?

elezar · 2025-06-27T11:36:15Z

container-toolkit/cdi-support.md

+$ sudo journalctl -u nvidia-cdi-refresh.service
+```
+
+### Manual CDI Specification Generation


Given the automatic process now writes to /var/run/cdi/nvidia.yaml we should update the instructions below to output to this file instead.

elezar · 2025-06-27T11:37:03Z

container-toolkit/cdi-support.md

@@ -77,6 +139,8 @@ You must generate a new CDI specification after any of the following changes:
 - You use a location such as `/var/run/cdi` that is cleared on boot.

 A configuration change can occur when MIG devices are created or removed, or when the driver is upgraded.
+
+**Note**: With NVIDIA Container Toolkit v1.18.0+, the automatic CDI refresh service handles most of these scenarios automatically.


Suggested change

**Note**: With NVIDIA Container Toolkit v1.18.0+, the automatic CDI refresh service handles most of these scenarios automatically.

**Note**: As of NVIDIA Container Toolkit v1.18.0, the automatic CDI refresh service handles most of these scenarios automatically.

a-mccarthy · 2025-06-27T15:41:44Z

container-toolkit/cdi-support.md

+
+### Manual CDI Specification Generation
+
+If you need to manually generate a CDI specification (for example, after MIG configuration changes or when using older versions), follow this procedure:


Suggested change

If you need to manually generate a CDI specification (for example, after MIG configuration changes or when using older versions), follow this procedure:

If you need to manually generate a CDI specification, for example, after MIG configuration changes or if your are using a Container Toolkit version before v1.18.0, follow this procedure:

just a nit to future proof the docs a bit.

a-mccarthy · 2025-06-27T15:43:14Z

container-toolkit/cdi-support.md

+# Add other nvidia-ctk environment variables as needed
+```
+
+For a complete list of available environment variables, refer to the `nvidia-ctk cdi generate` command documentation.


Is there a link to these docs?

a-mccarthy · 2025-06-27T15:44:46Z

container-toolkit/cdi-support.md

+# View service logs
+$ sudo journalctl -u nvidia-cdi-refresh.service
+```


+1

We typically split up the sample command and the example output so its clearer what folks need to run

Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>

Copilot

Pull Request Overview

This PR adds detailed documentation for the automatic and manual CDI specification generation via a new systemd service (nvidia-cdi-refresh), fixes typos in example workflows and release notes, and cleans up formatting in the install guide.

Introduces “Automatic CDI Specification Generation” with systemd path/service usage and customization instructions
Updates manual CDI generation path to /var/run/cdi/nvidia.yaml and revises related examples
Fixes spelling errors and removes an extraneous list marker

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
container-toolkit/sample-workload.md	Corrected typo in “configure”
container-toolkit/release-notes.md	Fixed “containers” spelling
container-toolkit/install-guide.md	Removed stray bullet
container-toolkit/cdi-support.md	Added systemd service documentation and updated paths

Comments suppressed due to low confidence (1)

container-toolkit/cdi-support.md:33

The PR title refers to nvidia-container-toolkit.service, but this section describes the nvidia-cdi-refresh units. Consider aligning the title and documentation to reference the correct service names or clarifying why they differ.

### Automatic CDI Specification Generation

ArangoGutierrez · 2025-07-01T07:46:01Z

@a-mccarthy / @elezar PTAL

ArangoGutierrez requested review from elezar and Copilot June 27, 2025 11:18

ArangoGutierrez self-assigned this Jun 27, 2025

This comment was marked as outdated.

Sign in to view

elezar reviewed Jun 27, 2025

View reviewed changes

ArangoGutierrez requested review from a-mccarthy and chenopis June 27, 2025 11:29

elezar reviewed Jun 27, 2025

View reviewed changes

ArangoGutierrez force-pushed the cdi_service branch from 0f836b0 to 62cb885 Compare June 27, 2025 12:01

ArangoGutierrez requested review from Copilot and elezar June 27, 2025 12:01

This comment was marked as outdated.

Sign in to view

a-mccarthy reviewed Jun 27, 2025

View reviewed changes

Add documentation for the systemd nvidia-container-toolkit.service

e3ff414

Signed-off-by: Carlos Eduardo Arango Gutierrez <eduardoa@nvidia.com>

ArangoGutierrez force-pushed the cdi_service branch from 62cb885 to e3ff414 Compare June 30, 2025 12:53

ArangoGutierrez requested a review from a-mccarthy June 30, 2025 12:53

ArangoGutierrez requested a review from Copilot June 30, 2025 12:53

Copilot AI reviewed Jun 30, 2025

View reviewed changes

	### Automatic CDI Specification Generation (v1.18.0+)
	### Automatic CDI Specification Generation

		# Manually trigger CDI generation
		$ sudo systemctl start nvidia-cdi-refresh.service

	Note: With NVIDIA Container Toolkit v1.18.0+, the automatic CDI refresh service handles most of these scenarios automatically.
	Note: As of NVIDIA Container Toolkit v1.18.0, the automatic CDI refresh service handles most of these scenarios automatically.


		### Manual CDI Specification Generation

		If you need to manually generate a CDI specification (for example, after MIG configuration changes or when using older versions), follow this procedure:

Add documentation for the systemd nvidia-container-toolkit.service #203

Are you sure you want to change the base?

Add documentation for the systemd nvidia-container-toolkit.service #203

Uh oh!

Conversation

ArangoGutierrez commented Jun 27, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

github-actions bot commented Jun 27, 2025

Documentation preview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

ArangoGutierrez commented Jul 1, 2025

Uh oh!

Uh oh!