Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for l4t Docker containers #117

Closed
wants to merge 1 commit into from

Conversation

j-baker
Copy link

@j-baker j-baker commented Jul 30, 2023

Fixes #5. I'm very new to Nix and so would appreciate a code style review if that's possible. This is a little closer to 'prototype' than 'good to go' but I figured it'd be worth pushing up regardless as others will likely benefit!

Description of changes

After this MR, setting virtualisation.docker.enableNvidia = true will work on Jetsons. Unfortunately there are two other flags which need handling:

  1. At time of writing the unified cgroup hierarchy is not supported by libnvidia-container, with the standard flags set. Therefore one must add systemd.enableUnifiedCgroupHierarchy = false or receive errors. I assume that with nixpkgs changes this will eventually fade.
  2. One must also disable the assertion assertion = cfg.enableNvidia -> config.hardware.opengl.driSupport32Bit or false which with a big stick could be done by assertions = lib.mkForce []. I have a PR for this here: nixos/virtualisation.docker: Do not assert 32 bit libraries available on ARM NixOS/nixpkgs#246179 but obviously it will be quite some time before it shows up in stable channels.
Testing

Tested on orin-nano. Tested by starting l4t-base, apt installing cuda, compiling samples, and running them (with success results).

@WatweA
Copy link

WatweA commented Nov 9, 2023

This PR merges master into your branch and has been tested builds on an Xavier NX - please take a look
j-baker#1

re: appears that there are issues running GPU docker containers, and using the nvidia runtime. However we can run base ubuntu and start up tegrastats as follows:

docker run --rm -v /nix/store:/nix/store -v /sys:/sys --privileged ubuntu /nix/store/...PATHTOBIN...-nvtools/bin/tegrastats

@jmbaur
Copy link
Contributor

jmbaur commented Nov 27, 2023

@j-baker would you be able to rebase this PR? A few files in the project got restructured pretty recently.

@jmbaur
Copy link
Contributor

jmbaur commented Dec 15, 2023

#172 is a successor to this PR. It uses NVIDIA's support for the CDI spec to declare what needs to be injected into the container. It also does not require disabling cgroups v2 on the host. It's been tested with a few different container images from nvcr.io/nvidia such as l4t-base and l4t-jetpack, but if there are any issues encountered, let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enable support for nvidia-container
4 participants