Add gke known issue #312
Merged
Add gke known issue #312
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Abigail McCarthy <20771501+a-mccarthy@users.noreply.github.com>
a-mccarthy
commented
Nov 14, 2025
| Refer to `GPU platforms <https://cloud.google.com/compute/docs/gpus>`_ | ||
| in the Google Cloud documentation. | ||
|
|
||
| .. note:: |
Collaborator
Author
There was a problem hiding this comment.
This is possibly not the best place for this note. open to suggestions :)
Documentation preview |
cdesiniotis
reviewed
Nov 14, 2025
Collaborator
Author
|
thanks for the review @cdesiniotis! can you take another look and approve? |
Co-authored-by: Christopher Desiniotis <chris.desiniotis@gmail.com> Signed-off-by: Abigail McCarthy <20771501+a-mccarthy@users.noreply.github.com>
cdesiniotis
approved these changes
Nov 17, 2025
Contributor
cdesiniotis
left a comment
cdesiniotis
left a comment
There was a problem hiding this comment.
Two minor suggestions, but otherwise lgtm. Feel free to merge this after addressing my comments.
|
|
||
| When installing NVIDIA GPU Operator on GKE 1.33+, there is a known issue where NVIDIA Container Toolkit will misconfigure the containerd `config.toml` file and prevent GPU Operator containers from starting up correctly. | ||
|
|
||
| To resolve this issue, set the ``RUNTIME_CONFIG_SOURCE=file`` environment variable in the toolkit container to resolve this issue. |
Contributor
There was a problem hiding this comment.
removing a small redundancy:
Suggested change
| To resolve this issue, set the ``RUNTIME_CONFIG_SOURCE=file`` environment variable in the toolkit container to resolve this issue. | |
| To resolve this issue, set the ``RUNTIME_CONFIG_SOURCE=file`` environment variable in the toolkit container. |
| Create the ConfigMap, then update the ClusterPolicy with the name of the configMap in the ``vgpuDeviceManager.config.name``, and restart the vgpu-device-manager pod. | ||
|
|
||
| - When using GKE 1.33+, there is a known issue where NVIDIA Container Toolkit will misconfigure the containerd `config.toml` file and prevent GPU Operator containers from starting up correctly. | ||
| To resolve this issue, set the ``RUNTIME_CONFIG_SOURCE=file`` environment variable in the toolkit container to resolve this issue. |
Contributor
There was a problem hiding this comment.
Suggested change
| To resolve this issue, set the ``RUNTIME_CONFIG_SOURCE=file`` environment variable in the toolkit container to resolve this issue. | |
| To resolve this issue, set the ``RUNTIME_CONFIG_SOURCE=file`` environment variable in the toolkit container. |
a-mccarthy
added a commit
to a-mccarthy/cloud-native-docs
that referenced
this pull request
Nov 18, 2025
* Add gke known issue Signed-off-by: Abigail McCarthy <20771501+a-mccarthy@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Christopher Desiniotis <chris.desiniotis@gmail.com> Signed-off-by: Abigail McCarthy <20771501+a-mccarthy@users.noreply.github.com> --------- Signed-off-by: Abigail McCarthy <20771501+a-mccarthy@users.noreply.github.com> Co-authored-by: Christopher Desiniotis <chris.desiniotis@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related to NVIDIA/gpu-operator#1577