EGL extensions not found in FreeBSD 12 (nvidia-driver 390.87_1)

[amshafer]

I have a quick question on some EGL extensions for the FreeBSD driver that can’t seem to be found. I’m posting here because the Nvidia forums for FreeBSD don’t seem to get any answers from developers and I found this issue trying to run this repository. If there is a better place for me to ask this question please let me know.

I am trying to write a small wayland compositor on FreeBSD, and while testing the eglstream example the FreeBSD driver reports that It can’t find certain extensions that patchnotes say are included.

Replication on FreeBSD 12.0-RELEASE amd64 on a GTX 1070 with nvidia-driver 390.87_1

(pkg install gmake …)
git clone https://github.com/aritger/eglstreams-kms-example
cd eglstreams-kms-example
gmake <——— Compilation with gmake works fine

running ./eglstreams-kms-example results in:
ERROR: eglGetProcAddress(eglGetOutputLayersEXT) failed

---

Some reading found that the extensions are searched for in the “utils.c” file. The 346 driver notes say:
Added support for the following EGL extensions:
EGL_EXT_device_base
EGL_EXT_platform_device
EGL_EXT_output_base <— It looks like this is the one missing

I don’t see EGL_EXT_output_base in the nvidia-settings app either. Is this repository a good way to get an OpenGL context without X11 on FreeBSD? And are these extensions still supported?

Please let me know if there is any more information I can provide, I am more than happy to help in any way to get this working, as it is a feature I am extremely interested in.

Thank you so much for your time!
Austin Shafer

[mvicomoya]

Hi @ashaferian,

These mechanisms were only enabled for FreeBSD very recently and drivers out of 390.xx won't include them.

Is there any chance you can try with 415.25?

[amshafer]

Hi @mvicomoya

Thanks for your reply! I have installed the 415.25 driver (didn't try it originally because it has yet to make it in the FreeBSD ports tree, I'll try to bump it to 415.25) but I still run into the following problem from line 154 of egl.c:

151     drmDeviceFile = pEglQueryDeviceStringEXT(device, EGL_DRM_DEVICE_FILE_EXT);
152 
153     if (drmDeviceFile == NULL) {
154         Fatal("No DRM device file found for EGL device.\n");
155     }

devfs does show the files /dev/nvidia-modeset /dev/nvidia0 /dev/nvidiactl. I am no expert on EGL, and certainly not on these extensions, but it seems like /dev/nvidia0 is what the drmDeviceFile is looking for?

On the plus side, it does seem to find the extensions that were originally missing. That's a start at least. Also I should mention that the 415.25 driver does work well on X11. Please let me know if there is anything I can do to help.
Thank you so much!

[aritger]

Sorry, we don't yet have DRM KMS support in the NVIDIA FreeBSD driver. So, the NVIDIA EGL implementation on FreeBSD won't be able to find a DRM device file.

[amshafer]

That is unfortunate to hear, though it certainly explains the problems I've had. Although I thought that's what the nvidia-modeset.ko module was supposed to provide? Or does it only support KMS without the full DRM feature set that EGL needs?

Is there any hope that this will be added in the future? I know most of the development time is spent on the Windows or Linux drivers but I'd really love to see this supported on FreeBSD.

Thanks again!

[aritger]

The core modesetting functionality is provided by nvidia-modeset.ko, but gluing that to the DRM APIs is provided on Linux by nvidia-drm.ko. We designed this with the intent of making nvidia-drm.ko available on FreeBSD, eventually, but that hasn't been done, yet. At the time we started on nvidia-drm.ko, the DRM infrastructure on FreeBSD was lagging behind Linux. I haven't kept up with recent FreeBSD DRM developments.

If you're interested and motivated, it might be possible for you to take the nvidia-drm.ko source from the Linux .run driver package, and port that to FreeBSD. Otherwise, it is on our todo list within NVIDIA, though I admit it may be difficult to prioritize very highly.

Sorry this isn't already in place.

[mvicomoya]

@ashaferian Yeah, what @aritger said. Sorry I misled you saying "these mechanisms were just enabled very recently". In fact, only some of these mechanisms were enabled. You should be able to use the Wayland EGL stack, but without a nvidia DRM KMS driver, EGL wouldn't be able to use EGLOutput to scanout.

Looking at https://github.com/FreeBSDDesktop/kms-drm, it might actually be relatively easy to grab nvidia-drm.ko source code from the Linux .run package (as @aritger also suggested) and use it as-is (or with very few modifications) using the linux-kpi compatibility interface.

[amshafer]

Thank you guys for pointing me in the right direction! I'd certainly be interested in giving it a try. I'll have to take a look at the linux nvidia-drm.ko and see if it can be plugged into the linuxkpi. Is all of nvidia-drm open source? If so I'd rather just port it to FreeBSD than rely on the linuxkpi. I'd need to read up on the FreeBSD DRM interface before either of these though.

Another quick question, is Vulkan supported? In the 390.xx drivers libEGL had the vkGetProcAddress symbol for the vulkan loader, but it seems to be removed in 415.25.

Thanks!

[amshafer]

Is all of nvidia-drm open source? If so I'd rather just port it to FreeBSD than rely on the linuxkpi. I'd need to read up on the FreeBSD DRM interface before either of these though.

Back at my desktop so I can answer part of this for myself, I do see all the sources in the nvidia-drm folder. Obviously the nvidia-drm-linux.c file is very linux specific, but how about the rest? some of the others like nvidia-dma-fence-helper.c seem to be linux specific as well but most don't appear to be. How closely tied are the remaining files? (i.e. is it a feasible port or would it involve a total rewrite?)

Thanks again for letting me pester you with questions!

[mvicomoya]

@ashaferian nvidia-drm sources are linux-centric overall. I don't think we have been very good at keeping linux-specific bits contained in nvidia-drm-linux.c, sorry.

I'm not a FreeBSD expert myself, but it seems the DRM APIs in FreeBSD and Linux differ considerably, so a port might entail more work than the linuxkpi approach.

As for your Vulkan question, we don't support Vulkan on FreeBSD. There have been changes related to GLVND, and that's probably the cause for vkGetProcAddress not being exported by libEGL, since now it's GLVND's.

[amshafer]

@mvicomoya Now that I think about it I recall seeing a mailing list post about how the linuxkpi based drm was what they were going to use going forward. I think the old drm you mentioned is getting removed next major release. I'll have to double check that but Linuxkpi seems the way to go.

So there wouldn't be any problems having nvidia-drm using the linuxkpi while nvidia.ko runs natively on FreeBSD? It sounds fine in theory but I figured I'd double check before I start investigating.

Ah that would make sense. Explains the removal of the symbol.

Thanks!

[amshafer]

Porting nvidia-drm to FreeBSD

Apologies for the long message

@mvicomoya @aritger Thanks for your advice from this thread, I have thrown some time into porting the nvidia-drm module using FreeBSD's linuxkpi and have some questions about a roadblock I have experienced. I'm sure I have made mistakes so please let me know if there is anything I can correct.

I placed what I have so far in a repo so that I can get feedback (and hopefully others can test). From reading Nvidia's license it seems like this was okay to do based off the linux/BSD exception. If this is not allowed, or is a problem in any way please let me know so that I can remove it.

• https://github.com/ashaferian/nvidia-driver

Current Status

nvidia-drm.ko builds and can be loaded on FreeBSD using the linuxkpi. /dev/dri/card0 is created and suspend/resume is not broken. This repo (eglstreams-kms-example) does not work however, as it 1) appears to need to link with libglvnd and 2) cannot perform drmModeGetResources. All my work was tested on a GTX 1070. A patch for pci id detection had to be added to the linuxkpi, so this port has to be run on FreeBSD 13.0-CURRENT revision 343451 or later.

Temporary hacks which need to be resolved at some point

• FreeBSD's drm port does not install the drm headers. They had to be copied into src/common/linuxkpi to get things to compile. Definitely not ideal. The FreeBSD drm devs said the nvidia-drm module should be added to their repo, but I doubt Nvidia would allow that. I'll have to sort out with them about getting those headers installed.
• src/nvidia-drm/nvidia-drm-conftest.h has the macro options enabled manually, as FreeBSD doesn't have conftest.sh in the nvidia driver.
• I don't know how to use libglvnd yet, and it appears this is the new way to get access to EGL? I will probably have to write a minimal program to link with libglvnd and then perform what this repo does.
• currently the linuxkpi seems to have a few memory leaks which are printed during the unloading of the nvidia-drm module. I don' t think they are nvidia-drm's fault but I plan on fixing them anyway.

The current problem (why I am writing to you)

One of the conftest options (NV_DRM_ATOMIC_MODESET_AVAILABLE) seems to be what actually enables most of the drm functionality. If this option is not enabled, nvidia-drm can be loaded successfully. It seems to work until the drmModeGetResources call, where it fails because this modeset option is not enabled.

I also had to implement some of the missing nvkms API functions in nvidia-modeset.ko. They appear to have been left out as nvidia-modeset originally didn't need them due to a lack of drm support.

The actual point of failure is line 303 of nvidia-drm-drv.c returning NULL
pDevice = nvKms->allocateDevice(&allocateDeviceParams);
The allocateDevice function pointer appears to point to _nv000007kms in the closed source portion of the driver, hence why I am a bit stuck.

• Is NV_DRM_ATOMIC_MODESET_AVAILABLE needed to realistically use drm?
• what is the best way to test if the drm portion works?
• What could be going wrong with the nvKms->allocateDevice call? It appears to fail after the nvkms_sema_alloc call.
• Should nvkms_open_from_kapi call nvidia_open_dev_kernel? That seems to be the FreeBSD driver's equivalent of the nvidia_open_common from the linux driver.

All in all there this port has some problems, and is far from perfect. It wasn't bad getting the open source portions ported, but now that the closed source allocation call is failing I could use a nudge in the right direction. I am still interested in getting this working so please tell me if there is anything I can do to resolve these issues and get this port functional. I can also provide a more in depth write up of what I actually had to change to get to this point, if it would be useful for future work on the FreeBSD driver.

Thank you so much for your time, I really appreciate your help.

Austin Shafer

[aritger]

Nice work, Austin :)

It is definitely fine to host your work in progess on github.

For the foreseeable future, I think we want to distribute the official nvidia-drm source as part of the NVIDIA driver package (we don't make guarantees about source or binary compatibility between NVIDIA driver components from different releases). So, yes, we'll need to find some compromise with the FreeBSD drm developers for access to drm header files.

libglvnd is a vendor-neutral dispatch layer for OpenGL, GLX, and EGL: https://github.com/NVIDIA/libglvnd AFAIK, it is enabled on FreeBSD. What specifically is the problem you're seeing with trying to use libglvnd?

Yes, NV_DRM_ATOMIC_MODESET_AVAILABLE is needed for nvidia-drm to support DRM Kernel Modesetting (KMS).

I'm sorry the code for nvKms->allocateDevice() is not currently publicly available. Reading through it, it will call back to nvidia-modeset-freebsd.c for:

nvkms_sema_alloc()
nvkms_open_gpu()
nvkms_call_rm()
nvkms_open_from_kapi()
nvkms_ioctl_from_kapi()

(and possibly others I missed)

It looks like you already have tracing code in nvidia-modeset-freebsd.c. Are any of those calls failing?

For your nvkms_open_from_kapi() question, I think the important part is that it calls nvKmsOpen(), which it looks like your nvidia-modeset-freebsd.c does.

For background:

• The nv-modeset-kernel.o binary is the core of the "NVKMS" ("NVIDIA modeset") driver, it is called mostly through its exported symbols nvKmsOpen(), nvKmsClose(), and nvKmsIoctl().
• Most clients of NVKMS are user-space driver components (The NVIDIA X driver, the NVIDIA OpenGL driver, etc), but nvidia-drm is a kernel-mode NVKMS client.
• nvidia-drm calls through the NVKMS "kapi" layer, which is part of the nv-modeset-kernel.o binary, which will then call through the nvidia-modeset-{linux,freebsd}.c "kernel interface layers", and then call into NVKMS through the same nvKmsOpen(), nvKmsClose(), and nvKmsIoctl() that user-space NVKMS clients use.

My guess at this point is that nvidia-modeset-freebsd.c is missing something that the NVKMS "kapi" layer is expecting. Try to trace all the nvidia-modeset-freebsd.c functions and see if any of those are failing.

If that still doesn't help shed light on things, then a next step could be to run nvidia-drm.ko on Linux, and trace the functions in nvidia-modeset-linux.c. I would expect the flow on nvidia-modeset-linux.c and nvidia-modeset-freebsd.c to match. Maybe that will help make it clearer what is different.

Sorry, that debugging may be a big tedious. Good luck!

[amshafer]

Thanks for your reply :)

So, yes, we'll need to find some compromise with the FreeBSD drm developers for access to drm header files.

I assume they don't install the headers because no one else uses them, so I think If I get this ported they should be open to having their port install them.

libglvnd AFAIK, it is enabled on FreeBSD. What specifically is the problem you're seeing with trying to use libglvnd?

No problems, I just wanted to make sure that was the proper way to get to EGL now. This repo wants to link with libEGL directly and fails, I wanted to make sure glvnd was the way to go before I spent time making a new minimal test app.

nvkms_call_rm()

Hmm I don't remember seeing or implementing this one. I'll have to add it. I don't think I get far enough for this to be the root cause, but its impossible to tell until I implement it. Do I need to do anything special to register these functions I'm adding to the kapi? I wonder if I missed a step and thats why my open function isn't being called.

It looks like you already have tracing code in nvidia-modeset-freebsd.c. Are any of those calls failing?

The only call that I see in /var/log/messages is to nvkms_sema_alloc, which returns a pointer to the lock. When poking around in the kernel debugger afterwards I can't read that address, so I think it may get freed during nvKms->allocateDevice failing? I assume that method has a 'failed' goto tag at the end like some of the other methods do.

For reference here is the debugging output from /var/log/messages:

Jan 29 22:52:05 wolfgang kernel: [drm] [nvidia-drm] nvKms:--------------
Jan 29 22:52:05 wolfgang kernel: [drm] [nvidia-drm] nvKms->enumerateGpus = ffffffff835545f0
Jan 29 22:52:05 wolfgang kernel: [drm] [nvidia-drm] nv_drm_driver.driver_features = 1f000
Jan 29 22:52:05 wolfgang kernel: drmn0: Comparing:--------------
Jan 29 22:52:05 wolfgang kernel: drmn0: vendor:10de == id->vendor:10de && device:1b81 == id->device:ffffffff
Jan 29 22:52:05 wolfgang kernel: drmn0: <drmn> on vgapci0
Jan 29 22:52:05 wolfgang kernel: drmn0: Comparing:--------------
Jan 29 22:52:05 wolfgang kernel: drmn0: vendor:10de == id->vendor:10de && device:1b81 == id->device:ffffffff
Jan 29 22:52:05 wolfgang kernel: [drm] [nvidia-drm] pci_dev fffff8003a76cc00 ------------------------
Jan 29 22:52:05 wolfgang kernel: [drm] [nvidia-drm] ->vendor = 4318
Jan 29 22:52:05 wolfgang kernel: [drm] [nvidia-drm] ->device = 65535
Jan 29 22:52:05 wolfgang kernel: [drm] [nvidia-drm] ->driver = ffffffff844d6170
Jan 29 22:52:05 wolfgang kernel: [drm] [nvidia-drm] [GPU ID 0x00000000] Loading driver
Jan 29 22:52:05 wolfgang kernel: nvkms_sema_alloc: creating mutex
Jan 29 22:52:05 wolfgang kernel: nvkms_sema_alloc: return 0x713f5860
Jan 29 22:52:05 wolfgang kernel: [drm:nv_drm_load] [nvidia-drm] [GPU ID 0x00000000] Failed to allocate NvKmsKapiDevice
Jan 29 22:52:05 wolfgang kernel: [drm:nv_drm_register_drm_device] [nvidia-drm] [GPU ID 0x00000000] Failed to register device
Jan 29 22:52:05 wolfgang kernel: device_attach: drmn0 attach returned -1
Jan 29 22:52:05 wolfgang kernel: [drm] [nvidia-drm] Registered pci driver with ret: 0

Its far from the greatest debugging output, but I can see the return from nvkms_sema_alloc (which should mean success). From reading the disassembled nvKms->allocateDevice It appears the next call is the nvkms_open_gpu, but I don't see any of my tracing print statements being called? That's about where I get lost.

Dump of assembler code for function _nv000007kms:
   0xffffffff83555be0 <+0>:	sub    $0x118,%rsp
   0xffffffff83555be7 <+7>:	mov    $0x1,%esi
   0xffffffff83555bec <+12>:	mov    %rbp,0xf0(%rsp)
   0xffffffff83555bf4 <+20>:	mov    %rdi,%rbp
   0xffffffff83555bf7 <+23>:	mov    $0xb8,%edi
   0xffffffff83555bfc <+28>:	mov    %rbx,0xe8(%rsp)
   0xffffffff83555c04 <+36>:	mov    %r12,0xf8(%rsp)
   0xffffffff83555c0c <+44>:	mov    %r13,0x100(%rsp)
   0xffffffff83555c14 <+52>:	mov    %r14,0x108(%rsp)
   0xffffffff83555c1c <+60>:	mov    %r15,0x110(%rsp)
   0xffffffff83555c24 <+68>:	callq  0xffffffff83515e20 <_nv002285kms>
   0xffffffff83555c29 <+73>:	test   %rax,%rax
   0xffffffff83555c2c <+76>:	mov    %rax,%rbx
   0xffffffff83555c2f <+79>:	je     0xffffffff83555c4b <_nv000007kms+107>
   0xffffffff83555c31 <+81>:	callq  0xffffffff83574ee0 <nvkms_sema_alloc>
   0xffffffff83555c36 <+86>:	test   %rax,%rax
   0xffffffff83555c39 <+89>:	mov    %rax,0x8(%rbx)
   0xffffffff83555c3d <+93>:	je     0xffffffff83555c4b <_nv000007kms+107>
   0xffffffff83555c3f <+95>:	mov    0x0(%rbp),%edi
   0xffffffff83555c42 <+98>:	callq  0xffffffff83574ae0 <nvkms_open_gpu> <---- It doesn't seem to get to here
...

For your nvkms_open_from_kapi() question, I think the important part is that it calls nvKmsOpen(), which it looks like your nvidia-modeset-freebsd.c does.

So should I use the nvkms_open_common that I wrote or the nvidia_open_dev_kernel? Or are you saying it shouldn't matter because they both call nvKmsOpen?

My guess at this point is that nvidia-modeset-freebsd.c is missing something that the NVKMS "kapi" layer is expecting. Try to trace all the nvidia-modeset-freebsd.c functions and see if any of those are failing.

I'll try adding print statements to every kapi call and see if I'm missing one. I only added statements to ones I added so I think your hunch is right and something is failing silently.

Sorry, that debugging may be a big tedious. Good luck!

Thanks! I suspect I'll have to pass my gpu through to a VM or something so I can step through it live. With my limited experience with this code its kind of hard to just look at the methods and tell whats going wrong.

Thanks again for all the help!

[aritger]

If nvkms_sema_alloc() returns non-NULL, then the next thing nvKms->allocateDevice() will do is call nvkms_open_gpu(). Can you add some tracing to it?

nvkms_call_rm() is already implemented in nvidia-modeset-freebsd.c. It is just that you may want to add tracing to it. But, I suppose its data is opaque, so there isn't much to be learned by tracing it.

For nvkms_open_from_kapi(), I think it will be best to pattern after the nvidia-modeset-linux.c version, which just calls nvkms_open_common()

[amshafer]

Yup and nvkms_call_rm is called super frequently, so it just seemed like noise when I traced it.

Made some progress, turns out I was only tracing the nvkms_open_from_kapi, however nvkms_open_gpu calls nvidia_open_dev_kernel instead. I have been using arbitrary GPU id's (as I couldn't find a way to get the correct id in the nvidia-drm code). This makes nvidia_open_dev_kernel fail while trying to get nvidia_find_state. I didn't realize until now that the main nvidia module used ids as well. I'll see if I can get the correct GPU id's later after class and hopefully that will solve this issue.

Cool I'll stick with the current linux modeled implementation of nvkms_open_common unless I run into problems with it.

Thanks!