internal/lookup/root Tests Fail in a Mock Chroot

[damaestro]

The following tests fail in a mock chroot environment:

github.com/NVIDIA/nvidia-container-toolkit/internal/lookup/root
--- FAIL: TestDriverLibrariesLocate (0.04s)
    --- FAIL: TestDriverLibrariesLocate/rootfs-no-cache-lib64libcuda.so.1 (0.00s)
        root_test.go:76: 
            	Error Trace:	/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/internal/lookup/root/root_test.go:76
            	Error:      	Not equal: 
            	            	expected: []string{"/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/testdata/lookup/rootfs-no-cache-lib64/usr/lib64/libcuda.so.999.88.77"}
            	            	actual  : []string{"/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/testdata/lookup/rootfs-no-cache-lib64/usr/lib64/libcuda.so.999.88.77"}
            	            	
            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -1,3 +1,3 @@
            	            	 ([]string) (len=1) {
            	            	- (string) (len=214) "/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/testdata/lookup/rootfs-no-cache-lib64/usr/lib64/libcuda.so.999.88.77"
            	            	+ (string) (len=160) "/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/testdata/lookup/rootfs-no-cache-lib64/usr/lib64/libcuda.so.999.88.77"
            	            	 }
            	Test:       	TestDriverLibrariesLocate/rootfs-no-cache-lib64libcuda.so.1
    --- FAIL: TestDriverLibrariesLocate/rootfs-no-cache-lib64libcuda.so.* (0.00s)
        root_test.go:76: 
            	Error Trace:	/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/internal/lookup/root/root_test.go:76
            	Error:      	Not equal: 
            	            	expected: []string{"/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/testdata/lookup/rootfs-no-cache-lib64/usr/lib64/libcuda.so.999.88.77"}
            	            	actual  : []string{"/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/testdata/lookup/rootfs-no-cache-lib64/usr/lib64/libcuda.so.999.88.77"}
            	            	
            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -1,3 +1,3 @@
            	            	 ([]string) (len=1) {
            	            	- (string) (len=214) "/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/testdata/lookup/rootfs-no-cache-lib64/usr/lib64/libcuda.so.999.88.77"
            	            	+ (string) (len=160) "/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/testdata/lookup/rootfs-no-cache-lib64/usr/lib64/libcuda.so.999.88.77"
            	            	 }
            	Test:       	TestDriverLibrariesLocate/rootfs-no-cache-lib64libcuda.so.*
    --- FAIL: TestDriverLibrariesLocate/rootfs-no-cache-lib64libcuda.so.*.* (0.00s)
        root_test.go:76: 
            	Error Trace:	/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/internal/lookup/root/root_test.go:76
            	Error:      	Not equal: 
            	            	expected: []string{"/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/testdata/lookup/rootfs-no-cache-lib64/usr/lib64/libcuda.so.999.88.77"}
            	            	actual  : []string{"/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/testdata/lookup/rootfs-no-cache-lib64/usr/lib64/libcuda.so.999.88.77"}
            	            	
            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -1,3 +1,3 @@
            	            	 ([]string) (len=1) {
            	            	- (string) (len=214) "/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/testdata/lookup/rootfs-no-cache-lib64/usr/lib64/libcuda.so.999.88.77"
            	            	+ (string) (len=160) "/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/testdata/lookup/rootfs-no-cache-lib64/usr/lib64/libcuda.so.999.88.77"
            	            	 }
            	Test:       	TestDriverLibrariesLocate/rootfs-no-cache-lib64libcuda.so.*.*
    --- FAIL: TestDriverLibrariesLocate/rootfs-no-cache-lib64libcuda.so.999.88.77 (0.00s)
        root_test.go:76: 
            	Error Trace:	/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/internal/lookup/root/root_test.go:76
            	Error:      	Not equal: 
            	            	expected: []string{"/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/testdata/lookup/rootfs-no-cache-lib64/usr/lib64/libcuda.so.999.88.77"}
            	            	actual  : []string{"/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/testdata/lookup/rootfs-no-cache-lib64/usr/lib64/libcuda.so.999.88.77"}
            	            	
            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -1,3 +1,3 @@
            	            	 ([]string) (len=1) {
            	            	- (string) (len=214) "/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/testdata/lookup/rootfs-no-cache-lib64/usr/lib64/libcuda.so.999.88.77"
            	            	+ (string) (len=160) "/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/testdata/lookup/rootfs-no-cache-lib64/usr/lib64/libcuda.so.999.88.77"
            	            	 }
            	Test:       	TestDriverLibrariesLocate/rootfs-no-cache-lib64libcuda.so.999.88.77
    --- FAIL: TestDriverLibrariesLocate/rootfs-1libcuda.so.1 (0.00s)
        root_test.go:76: 
            	Error Trace:	/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/internal/lookup/root/root_test.go:76
            	Error:      	Not equal: 
            	            	expected: []string{"/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu/libcuda.so.999.88.77"}
            	            	actual  : []string{"/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu/libcuda.so.999.88.77"}
            	            	
            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -1,3 +1,3 @@
            	            	 ([]string) (len=1) {
            	            	- (string) (len=212) "/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu/libcuda.so.999.88.77"
            	            	+ (string) (len=158) "/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu/libcuda.so.999.88.77"
            	            	 }
            	Test:       	TestDriverLibrariesLocate/rootfs-1libcuda.so.1
    --- FAIL: TestDriverLibrariesLocate/rootfs-1libcuda.so.* (0.01s)
        root_test.go:76: 
            	Error Trace:	/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/internal/lookup/root/root_test.go:76
            	Error:      	Not equal: 
            	            	expected: []string{"/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu/libcuda.so.999.88.77"}
            	            	actual  : []string{"/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu/libcuda.so.999.88.77"}
            	            	
            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -1,3 +1,3 @@
            	            	 ([]string) (len=1) {
            	            	- (string) (len=212) "/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu/libcuda.so.999.88.77"
            	            	+ (string) (len=158) "/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu/libcuda.so.999.88.77"
            	            	 }
            	Test:       	TestDriverLibrariesLocate/rootfs-1libcuda.so.*
    --- FAIL: TestDriverLibrariesLocate/rootfs-1libcuda.so.*.* (0.00s)
        root_test.go:76: 
            	Error Trace:	/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/internal/lookup/root/root_test.go:76
            	Error:      	Not equal: 
            	            	expected: []string{"/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu/libcuda.so.999.88.77"}
            	            	actual  : []string{"/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu/libcuda.so.999.88.77"}
            	            	
            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -1,3 +1,3 @@
            	            	 ([]string) (len=1) {
            	            	- (string) (len=212) "/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu/libcuda.so.999.88.77"
            	            	+ (string) (len=158) "/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu/libcuda.so.999.88.77"
            	            	 }
            	Test:       	TestDriverLibrariesLocate/rootfs-1libcuda.so.*.*
    --- FAIL: TestDriverLibrariesLocate/rootfs-1libcuda.so.999.88.77 (0.00s)
        root_test.go:76: 
            	Error Trace:	/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/internal/lookup/root/root_test.go:76
            	Error:      	Not equal: 
            	            	expected: []string{"/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu/libcuda.so.999.88.77"}
            	            	actual  : []string{"/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu/libcuda.so.999.88.77"}
            	            	
            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -1,3 +1,3 @@
            	            	 ([]string) (len=1) {
            	            	- (string) (len=212) "/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/_build/src/github.com/NVIDIA/nvidia-container-toolkit/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu/libcuda.so.999.88.77"
            	            	+ (string) (len=158) "/builddir/build/BUILD/nvidia-container-toolkit-1.17.1-build/nvidia-container-toolkit-1.17.1/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu/libcuda.so.999.88.77"
            	            	 }
            	Test:       	TestDriverLibrariesLocate/rootfs-1libcuda.so.999.88.77
FAIL
exit status 1
FAIL	github.com/NVIDIA/nvidia-container-toolkit/internal/lookup/root	0.043s

[elezar]

@damaestro could you please provide more information on how you're running the tests?

[damaestro]

@elezar This is for a build in the Fedora build tools. Specifically, this is a build in mock. It's a fancy chroot build tool for reproducible builds.

If you'd like to reproduce, you can use mock with the source from https://copr.fedorainfracloud.org/coprs/g/ai-ml/nvidia-container-toolkit/build/8243448/ and update the %check to not filter the test. We haven't updated the main distro package yet, waiting to figure this out.

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[elezar]

@damaestro were you able to solve this problem?

Note that we don't have much experience with this particular build environment. Would you be able to provide a containerized reproducer for this so that we can debug it if it is still relevant?

ALso, I noticed that the builds still do not include v1.18.0. Is this a result of the failing build?

[damaestro]

@elezar No, there are still issues.

I've not done the latest update due to new testing issues (maybe related) and a lacking test skipping ability without patches.

The test that is failing when I'm using our %gocheck macro now is:

+ GO_LDFLAGS=' -X github.com/NVIDIA/nvidia-container-toolkit/version=1.18.0'
+ GO_TEST_FLAGS='-buildmode pie -compiler gc'
+ GO_TEST_EXT_LD_FLAGS='-Wl,-z,relro -Wl,--as-needed  -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-hardened-ld-errors -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes   -Wl,-z,lazy -Wl,--export-dynamic'
+ go-rpm-integration check -i github.com/NVIDIA/nvidia-container-toolkit -b /builddir/build/BUILD/nvidia-container-toolkit-1.18.0-build/nvidia-container-toolkit-1.18.0/_build/bin -s /builddir/build/BUILD/nvidia-container-toolkit-1.18.0-build/nvidia-container-toolkit-1.18.0/_build -V 1.18.0-1.fc44 -p /builddir/build/BUILD/nvidia-container-toolkit-1.18.0-build/BUILDROOT -g /usr/share/gocode -r '.*example.*' ./...
Testing    in: /builddir/build/BUILD/nvidia-container-toolkit-1.18.0-build/nvidia-container-toolkit-1.18.0/_build/src
         PATH: /builddir/build/BUILD/nvidia-container-toolkit-1.18.0-build/nvidia-container-toolkit-1.18.0/_build/bin:/usr/bin:/bin:/usr/sbin:/sbin
       GOPATH: /builddir/build/BUILD/nvidia-container-toolkit-1.18.0-build/nvidia-container-toolkit-1.18.0/_build:/usr/share/gocode
  GO111MODULE: off
      command: go test -buildmode pie -compiler gc -ldflags " -X github.com/NVIDIA/nvidia-container-toolkit/version=1.18.0 -extldflags '-Wl,-z,relro -Wl,--as-needed  -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-hardened-ld-errors -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes   -Wl,-z,lazy -Wl,--export-dynamic'"
[...]
--- FAIL: TestInstall (0.03s)
    --- FAIL: TestInstall/#03 (0.02s)
        toolkit_test.go:209: 
            	Error Trace:	/builddir/build/BUILD/nvidia-container-toolkit-1.18.0-build/nvidia-container-toolkit-1.18.0/_build/src/github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk-installer/toolkit/toolkit_test.go:209
            	Error:      	Not equal: 
            	            	expected: "---\ncdiVersion: 0.5.0\nkind: example.com/class\ndevices:\n    - name: all\n      containerEdits:\n        deviceNodes:\n            - path: /dev/nvidia0\n              hostPath: /host/driver/root/dev/nvidia0\n            - path: /dev/nvidiactl\n              hostPath: /host/driver/root/dev/nvidiactl\n            - path: /dev/nvidia-caps-imex-channels/channel0\n              hostPath: /host/driver/root/dev/nvidia-caps-imex-channels/channel0\n            - path: /dev/nvidia-caps-imex-channels/channel1\n              hostPath: /host/driver/root/dev/nvidia-caps-imex-channels/channel1\n            - path: /dev/nvidia-caps-imex-channels/channel2047\n              hostPath: /host/driver/root/dev/nvidia-caps-imex-channels/channel2047\n            - path: /dev/nvidia-caps/nvidia-cap1\n              hostPath: /host/driver/root/dev/nvidia-caps/nvidia-cap1\ncontainerEdits:\n    env:\n        - NVIDIA_CTK_LIBCUDA_DIR=/lib/x86_64-linux-gnu\n        - NVIDIA_VISIBLE_DEVICES=void\n    hooks:\n        - hookName: createContainer\n          path: /tmp/TestInstall#03950651923/001/toolkit-test/nvidia-cdi-hook\n          args:\n            - nvidia-cdi-hook\n            - create-symlinks\n            - --link\n            - libcuda.so.1::/lib/x86_64-linux-gnu/libcuda.so\n          env:\n            - NVIDIA_CTK_DEBUG=false\n        - hookName: createContainer\n          path: /tmp/TestInstall#03950651923/001/toolkit-test/nvidia-cdi-hook\n          args:\n            - nvidia-cdi-hook\n            - update-ldcache\n            - --folder\n            - /lib/x86_64-linux-gnu\n            - --folder\n            - /lib/x86_64-linux-gnu/vdpau\n          env:\n            - NVIDIA_CTK_DEBUG=false\n    mounts:\n        - hostPath: /host/driver/root/lib/x86_64-linux-gnu/libcuda.so.999.88.77\n          containerPath: /lib/x86_64-linux-gnu/libcuda.so.999.88.77\n          options:\n            - ro\n            - nosuid\n            - nodev\n            - rbind\n            - rprivate\n        - hostPath: /host/driver/root/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.999.88.77\n          containerPath: /lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.999.88.77\n          options:\n            - ro\n            - nosuid\n            - nodev\n            - rbind\n            - rprivate\n"
            	            	actual  : "---\ncdiVersion: 0.5.0\nkind: example.com/class\ndevices:\n    - name: all\n      containerEdits:\n        deviceNodes:\n            - path: /dev/nvidia0\n              hostPath: /host/driver/root/dev/nvidia0\n            - path: /dev/nvidiactl\n              hostPath: /host/driver/root/dev/nvidiactl\n            - path: /dev/nvidia-caps-imex-channels/channel0\n              hostPath: /host/driver/root/dev/nvidia-caps-imex-channels/channel0\n            - path: /dev/nvidia-caps-imex-channels/channel1\n              hostPath: /host/driver/root/dev/nvidia-caps-imex-channels/channel1\n            - path: /dev/nvidia-caps-imex-channels/channel2047\n              hostPath: /host/driver/root/dev/nvidia-caps-imex-channels/channel2047\n            - path: /dev/nvidia-caps/nvidia-cap1\n              hostPath: /host/driver/root/dev/nvidia-caps/nvidia-cap1\ncontainerEdits:\n    env:\n        - NVIDIA_CTK_LIBCUDA_DIR=/builddir/build/BUILD/nvidia-container-toolkit-1.18.0-build/nvidia-container-toolkit-1.18.0/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu\n        - NVIDIA_VISIBLE_DEVICES=void\n    hooks:\n        - hookName: createContainer\n          path: /tmp/TestInstall#03950651923/001/toolkit-test/nvidia-cdi-hook\n          args:\n            - nvidia-cdi-hook\n            - update-ldcache\n          env:\n            - NVIDIA_CTK_DEBUG=false\n"
            	            	
            	            	Diff:
            	            	--- Expected
            	            	+++ Actual
            	            	@@ -21,3 +21,3 @@
            	            	     env:
            	            	-        - NVIDIA_CTK_LIBCUDA_DIR=/lib/x86_64-linux-gnu
            	            	+        - NVIDIA_CTK_LIBCUDA_DIR=/builddir/build/BUILD/nvidia-container-toolkit-1.18.0-build/nvidia-container-toolkit-1.18.0/testdata/lookup/rootfs-1/lib/x86_64-linux-gnu
            	            	         - NVIDIA_VISIBLE_DEVICES=void
            	            	@@ -28,35 +28,5 @@
            	            	             - nvidia-cdi-hook
            	            	-            - create-symlinks
            	            	-            - --link
            	            	-            - libcuda.so.1::/lib/x86_64-linux-gnu/libcuda.so
            	            	+            - update-ldcache
            	            	           env:
            	            	             - NVIDIA_CTK_DEBUG=false
            	            	-        - hookName: createContainer
            	            	-          path: /tmp/TestInstall#03950651923/001/toolkit-test/nvidia-cdi-hook
            	            	-          args:
            	            	-            - nvidia-cdi-hook
            	            	-            - update-ldcache
            	            	-            - --folder
            	            	-            - /lib/x86_64-linux-gnu
            	            	-            - --folder
            	            	-            - /lib/x86_64-linux-gnu/vdpau
            	            	-          env:
            	            	-            - NVIDIA_CTK_DEBUG=false
            	            	-    mounts:
            	            	-        - hostPath: /host/driver/root/lib/x86_64-linux-gnu/libcuda.so.999.88.77
            	            	-          containerPath: /lib/x86_64-linux-gnu/libcuda.so.999.88.77
            	            	-          options:
            	            	-            - ro
            	            	-            - nosuid
            	            	-            - nodev
            	            	-            - rbind
            	            	-            - rprivate
            	            	-        - hostPath: /host/driver/root/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.999.88.77
            	            	-          containerPath: /lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.999.88.77
            	            	-          options:
            	            	-            - ro
            	            	-            - nosuid
            	            	-            - nodev
            	            	-            - rbind
            	            	-            - rprivate
            	            	 
            	Test:       	TestInstall/#03
FAIL
exit status 1
FAIL	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk-installer/toolkit	0.034s
error: Bad exit status from /var/tmp/rpm-tmp.ouyvXc (%check)
    Bad exit status from /var/tmp/rpm-tmp.ouyvXc (%check)

And if I use the new macro %gotest I end up with:

+ go test -buildmode pie -compiler gc -ldflags ' -X github.com/NVIDIA/nvidia-container-toolkit/version=1.18.0 -extldflags '\''-Wl,-z,relro -Wl,--as-needed  -Wl,-z,pack-relative-relocs -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-hardened-ld-errors -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -Wl,--build-id=sha1 -specs=/usr/lib/rpm/redhat/redhat-package-notes   -Wl,-z,lazy -Wl,--export-dynamic'\''' ./...
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-cdi-hook	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-cdi-hook/chmod	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-cdi-hook/commands	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-cdi-hook/create-symlinks	0.008s
ok  	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-cdi-hook/cudacompat	0.008s
ok  	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-cdi-hook/disable-device-node-modification	0.004s
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-cdi-hook/update-ldcache	[no test files]
--- FAIL: TestGoodInput (0.00s)
    main_test.go:110: executing: nvidia-container-runtime run --bundle /builddir/build/BUILD/nvidia-container-toolkit-1.18.0-build/nvidia-container-toolkit-1.18.0/tests/output/bundle testcontainer
    main_test.go:112: 
        	Error Trace:	/builddir/build/BUILD/nvidia-container-toolkit-1.18.0-build/nvidia-container-toolkit-1.18.0/cmd/nvidia-container-runtime/main_test.go:112
        	Error:      	Received unexpected error:
        	            	exec: "nvidia-container-runtime": executable file not found in $PATH
        	Test:       	TestGoodInput
        	Messages:   	runtime should not return an error%!(EXTRA string=output=%v, string=)
--- FAIL: TestDuplicateHook (0.00s)
    main_test.go:144: inserting nvidia prestart hook to config.json
    main_test.go:166: executing: nvidia-container-runtime create --bundle /builddir/build/BUILD/nvidia-container-toolkit-1.18.0-build/nvidia-container-toolkit-1.18.0/tests/output/bundle testcontainer
    main_test.go:168: 
        	Error Trace:	/builddir/build/BUILD/nvidia-container-toolkit-1.18.0-build/nvidia-container-toolkit-1.18.0/cmd/nvidia-container-runtime/main_test.go:168
        	Error:      	Received unexpected error:
        	            	exec: "nvidia-container-runtime": executable file not found in $PATH
        	Test:       	TestDuplicateHook
        	Messages:   	runtime should not return an error%!(EXTRA string=output=%v, string=)
FAIL
FAIL	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-container-runtime	0.008s
ok  	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-container-runtime-hook	0.006s
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-container-runtime.cdi	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-container-runtime.legacy	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk/cdi	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk/cdi/generate	0.100s
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk/cdi/list	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk/cdi/transform	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk/cdi/transform/root	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk/config	0.004s
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk/config/create-default	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk/config/flags	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk/hook	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk/info	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk/runtime	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk/runtime/configure	0.013s
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk/system	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk/system/create-dev-char-symlinks	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk/system/create-device-nodes	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk-installer	0.083s
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk-installer/container	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk-installer/container/operator	0.003s
?   	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk-installer/container/runtime	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk-installer/container/runtime/containerd	0.018s
ok  	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk-installer/container/runtime/crio	0.009s
ok  	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk-installer/container/runtime/docker	0.005s
ok  	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk-installer/toolkit	0.039s
ok  	github.com/NVIDIA/nvidia-container-toolkit/cmd/nvidia-ctk-installer/toolkit/installer	0.005s
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/config	0.006s
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/config/image	0.006s
?   	github.com/NVIDIA/nvidia-container-toolkit/internal/cuda	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/discover	0.006s
?   	github.com/NVIDIA/nvidia-container-toolkit/internal/dxcore	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/edits	0.004s
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/info	0.006s
?   	github.com/NVIDIA/nvidia-container-toolkit/internal/info/drm	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/internal/info/proc	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/info/proc/devices	0.005s
?   	github.com/NVIDIA/nvidia-container-toolkit/internal/ldcache	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/internal/ldconfig	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/internal/logger	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/lookup	0.086s
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/lookup/root	0.049s
?   	github.com/NVIDIA/nvidia-container-toolkit/internal/lookup/symlinks	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/modifier	0.007s
?   	github.com/NVIDIA/nvidia-container-toolkit/internal/modifier/cdi	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/nvcaps	0.005s
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/nvsandboxutils	0.006s
?   	github.com/NVIDIA/nvidia-container-toolkit/internal/nvsandboxutils/gen/nvsandboxutils	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/internal/nvsandboxutils/mock	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/oci	0.010s
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/platform-support/dgpu	0.006s
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/platform-support/tegra	0.006s
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/platform-support/tegra/csv	0.006s
?   	github.com/NVIDIA/nvidia-container-toolkit/internal/requirements	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/requirements/constraints	0.006s
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/runtime	0.011s
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/system/nvdevices	0.005s
ok  	github.com/NVIDIA/nvidia-container-toolkit/internal/system/nvmodules	0.005s
?   	github.com/NVIDIA/nvidia-container-toolkit/internal/test	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/pkg/config	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/pkg/config/engine	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/pkg/config/engine/containerd	0.008s
ok  	github.com/NVIDIA/nvidia-container-toolkit/pkg/config/engine/crio	0.006s
ok  	github.com/NVIDIA/nvidia-container-toolkit/pkg/config/engine/docker	0.005s
?   	github.com/NVIDIA/nvidia-container-toolkit/pkg/config/ocihook	[no test files]
?   	github.com/NVIDIA/nvidia-container-toolkit/pkg/config/toml	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/pkg/nvcdi	0.012s
ok  	github.com/NVIDIA/nvidia-container-toolkit/pkg/nvcdi/spec	0.006s
ok  	github.com/NVIDIA/nvidia-container-toolkit/pkg/nvcdi/transform	0.004s
?   	github.com/NVIDIA/nvidia-container-toolkit/pkg/nvcdi/transform/noop	[no test files]
ok  	github.com/NVIDIA/nvidia-container-toolkit/pkg/nvcdi/transform/root	0.004s
FAIL
error: Bad exit status from /var/tmp/rpm-tmp.2fpWzO (%check)
    Bad exit status from /var/tmp/rpm-tmp.2fpWzO (%check)

I've not had time to track down how best to evaluate and disable (or report) the test failures.