Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix tf model by upgrading driver version #1557

Merged
merged 8 commits into from Dec 21, 2023
Merged

Fix tf model by upgrading driver version #1557

merged 8 commits into from Dec 21, 2023

Conversation

ashmeenkaur
Copy link
Collaborator

@ashmeenkaur ashmeenkaur commented Dec 21, 2023 •
edited

Description

Tensorflow model was failing with the following error:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: requirement error: unsatisfied condition: cuda>=11.8, please update your driver to a newer version, or use an earlier cuda container: unknown.

Fix: Upgraded driver version to 520.61.05 based on documentation: https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cuda-major-component-versions

Also updated VM image as ubuntu-2004-focal-v20230616 was deprecated.

Link to the issue in case of a bug fix.

NA

Testing details

  1. Manual - Manually ran the model with the changes and it is working now.
  2. Unit tests - NA
  3. Integration tests - NA

@ashmeenkaur ashmeenkaur marked this pull request as ready for review December 21, 2023 12:08
@ashmeenkaur ashmeenkaur merged commit 90a4927 into master Dec 21, 2023
8 checks passed
@ashmeenkaur ashmeenkaur deleted the fix-tf-model branch December 22, 2023 07:02
@chiefmikey chiefmikey mentioned this pull request Dec 29, 2023
Merged
gargnitingoogle pushed a commit that referenced this pull request Jan 4, 2024
@ashmeenkaur @gargnitingoogle
Fix tf model by upgrading driver version (#1557)
133ddfe 
* change dependencies to use CUDA 11.8

* temp commit

* upgrade ubuntu image as old is deprecated

* reduce sleep

* upgrade driver version

* revert to newer version

* Revert "reduce sleep"

This reverts commit 69828ba.

* Revert "temp commit"

This reverts commit d1cce7f.
ashmeenkaur added a commit that referenced this pull request Jan 9, 2024
@ashmeenkaur
Fix tf model by upgrading driver version (#1557)
1727ed2 
* change dependencies to use CUDA 11.8

* temp commit

* upgrade ubuntu image as old is deprecated

* reduce sleep

* upgrade driver version

* revert to newer version

* Revert "reduce sleep"

This reverts commit 69828ba.

* Revert "temp commit"

This reverts commit d1cce7f.
ashmeenkaur added a commit that referenced this pull request Jan 10, 2024
@ashmeenkaur
Fix tf model by upgrading driver version (#1557)
826426b 
* change dependencies to use CUDA 11.8

* temp commit

* upgrade ubuntu image as old is deprecated

* reduce sleep

* upgrade driver version

* revert to newer version

* Revert "reduce sleep"

This reverts commit 69828ba.

* Revert "temp commit"

This reverts commit d1cce7f.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sethiay sethiay sethiay approved these changes

@vadlakondaswetha vadlakondaswetha Awaiting requested review from vadlakondaswetha

@raj-prince raj-prince Awaiting requested review from raj-prince

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@ashmeenkaur @sethiay