oci-nvidia-hook not working as expected with podman command

[garyedwards]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

Description

Steps to reproduce the issue:

Set up the the hook as per the the Openshift guide: https://blog.openshift.com/use-gpus-with-device-plugin-in-openshift-3-9/

I have also tried the 1.0.0 Hook Schema Nvidia example documented in this repo for the oci-nvidia-hook.json file.

Run the test container suggested in the Openshift guide with podman:

sudo podman run -it --rm docker.io/mirrorgooglecontainers/cuda-vector-add:v0.1

Describe the results you received:
None of the Nvidia or CUDA tools are mounted and the test fails:

Failed to allocate device vector A (error code CUDA driver version is insufficient for CUDA runtime version)!

Describe the results you expected:
For the vector-add test to pass as it does when using the fedora docker command.

Additional information you deem important (e.g. issue happens only occasionally):
A work around is to use the nvidia-container-runtime with the podman --runtime option but I would rather use the native runc with the Nvidia hook if possible.

Output of podman version:

Version:       0.6.1-dev
Go Version:    go1.10.2
OS/Arch:       linux/amd64

Output of podman info:

host:
  MemFree: 181026816
  MemTotal: 8314044416
  SwapFree: 8455450624
  SwapTotal: 8455712768
  arch: amd64
  cpus: 8
  hostname: kfworkstation
  kernel: 4.16.12-200.fc27.x86_64
  os: linux
  uptime: 4h 29m 19.44s (Approximately 0.17 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
store:
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
  - overlay.override_kernel_check=true
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
  ImageStore:
    number: 13
  RunRoot: /var/run/containers/storage

Additional environment details (AWS, VirtualBox, physical, etc.):
Running on physical hardware with Quadro 1000M.

[mheon]

@garyedwards Can you add a --log-level=debug to the podman run command you're using and pastebin the results?

I know hooks in general are working, so I suspect that the hook in question isn't being added. This likely comes down to a missing annotation which would be used to identify the container as requiring the nvidia hook.

[garyedwards]

@mheon see link to pastebin below as requested. I have tried using the "when" "always" from the 1.0.0 Schema as well as annotations to force it to trigger, also with no success. Any advice much appreciated.

https://pastebin.com/t9HABDKQ

[rhatdan]

What does the the file in /usr/share/containers/oci/hooks.d look like?

[garyedwards]

Hi @rhatdan, see the contents below:

{
   "hook": "/usr/bin/nvidia-container-runtime-hook",
   "stage": [ "prestart" ]
}

I have also tried the various versions from the man page in this repo. The above works with the fedora docker command.

Docker version 1.13.1, build 19d432b-unsupported

[rhatdan]

You need to add an Annotations flag to your config and then add the Annotations to the container.
"annotations": {"useNvidia"}

And then run a container with tue UseNvidia annotation. Not fully sure of the syntax. @wking probably would no best.

[garyedwards]

Excellent so the follow works great as "sandbox" seems to be a default annotation. Now will look into getting the buildah bud --annotation option adding annotations so that it does not run for every container. Initial tests are not adding annotations visible in podman inspect command, but I will look deeper and open a ticket on buildah if it seems to be an issue.

Thanks both for your help!

{
   "hook": "/usr/bin/nvidia-container-runtime-hook",
   "arguments": ["prestart"],
   "annotations": ["sandbox"],
   "stage": [ "prestart" ]
}

[garyedwards]

Just a quick not on the annotations. Buildah seems to add the annotations to the image which can be seen with a podman inspect agaist that image.

However but when podman creates a new container and that container is inspected the annotations are lost and only a load of default io.kubernetes.cri-o annotations are present. I am not sure if this is the correct behaviour or if my image annotations should be carried over to the container? Will keep digging.

[mheon]

@garyedwards I think that's a bug. Took a quick look at the code, and it doesn't we don't actually preserve the annotations when creating a container from an image. Can you make an issue about this to track it? Should be an easy fix on our end.

I wonder if it's also worth it to add a flag for modifying annotations of the created container.

[wking]

@garyedwards, I'm surprised the 1.0.0 hooks did not work. You tried this example? Which annotation key has the sandbox value? And I'm really surprised it didn't fire when you set always true.

[garyedwards]

Hi @wking, the 1.0.0 may well work now that I have got to the bottom of this a bit further. I have just opened a second issue #885 relating to annotations not being carried across to containers. This may have been making the 1.0.0 hooks appear not to work in my tests when in fact it was due to the annotation not being there. I will test with the 1.0.0 syntax again shortly. The sandbox key is below:

"io.kubernetes.cri-o.ContainerType": "sandbox" 

[wking]

So try:

"when": {
  "annotations": {
    "^io\\.kubernetes\\.cri-o\\.ContainerType$": "^sandbox$"
  }
}

(looks like I need to double the backslashes in the docs).

But annotations not getting passed through doesn't explain why:

"when": {
  "always": true
}

didn't work for you earlier.

[garyedwards]

I have tried the following two versions both of which result in the command hanging. The third 0.1.0 style works as expected. See pastebin for the debug log from 1.0.0 syntax: https://pastebin.com/xkFqP9Js

{
  "hook": {
    "path": "/usr/bin/nvidia-container-runtime-hook",
    "args": ["nvidia-container-runtime-hook", "prestart"]
  },
  "when": {
    "annotations": {
      "^io\\.kubernetes\\.cri-o\\.ContainerType$": "^sandbox$"
    }
  },
  "stages": ["prestart"]
}

Always true version:

{
  "hook": {
    "path": "/usr/bin/nvidia-container-runtime-hook",
    "args": ["nvidia-container-runtime-hook", "prestart"]
  },
  "when": {
    "always": true
  },
  "stages": ["prestart"]
}

Working 0.1.0 version:

{
   "hook": "/usr/bin/nvidia-container-runtime-hook",
   "arguments": ["prestart"],
   "annotations": ["sandbox"],
   "stage": [ "prestart" ]
}

[wking]

time="2018-06-03T16:32:17+01:00" level=warning msg="failed to parse language "en_GB.UTF-8": language: tag is not well-formed" 

That is golang/go#25340, but you should be getting our POSIX fallback. Both of those are followed by "Cleaning up container" messages, so I don't see anything obvious going wrong (but no evidence of them going right either).

[mheon]

@wking I think it might help to add some debug logging to the hooks package, so we can see what hooks it registers and what hooks it adds