EGL extensions not found in FreeBSD 12 (nvidia-driver 390.87_1) #7
Comments
|
Hi @ashaferian, These mechanisms were only enabled for FreeBSD very recently and drivers out of 390.xx won't include them. Is there any chance you can try with 415.25? |
|
Hi @mvicomoya Thanks for your reply! I have installed the 415.25 driver (didn't try it originally because it has yet to make it in the FreeBSD ports tree, I'll try to bump it to 415.25) but I still run into the following problem from line 154 of egl.c: devfs does show the files /dev/nvidia-modeset /dev/nvidia0 /dev/nvidiactl. I am no expert on EGL, and certainly not on these extensions, but it seems like /dev/nvidia0 is what the drmDeviceFile is looking for? On the plus side, it does seem to find the extensions that were originally missing. That's a start at least. Also I should mention that the 415.25 driver does work well on X11. Please let me know if there is anything I can do to help. |
|
Sorry, we don't yet have DRM KMS support in the NVIDIA FreeBSD driver. So, the NVIDIA EGL implementation on FreeBSD won't be able to find a DRM device file. |
|
That is unfortunate to hear, though it certainly explains the problems I've had. Although I thought that's what the nvidia-modeset.ko module was supposed to provide? Or does it only support KMS without the full DRM feature set that EGL needs? Is there any hope that this will be added in the future? I know most of the development time is spent on the Windows or Linux drivers but I'd really love to see this supported on FreeBSD. Thanks again! |
|
The core modesetting functionality is provided by nvidia-modeset.ko, but gluing that to the DRM APIs is provided on Linux by nvidia-drm.ko. We designed this with the intent of making nvidia-drm.ko available on FreeBSD, eventually, but that hasn't been done, yet. At the time we started on nvidia-drm.ko, the DRM infrastructure on FreeBSD was lagging behind Linux. I haven't kept up with recent FreeBSD DRM developments. If you're interested and motivated, it might be possible for you to take the nvidia-drm.ko source from the Linux .run driver package, and port that to FreeBSD. Otherwise, it is on our todo list within NVIDIA, though I admit it may be difficult to prioritize very highly. Sorry this isn't already in place. |
|
@ashaferian Yeah, what @aritger said. Sorry I misled you saying "these mechanisms were just enabled very recently". In fact, only some of these mechanisms were enabled. You should be able to use the Wayland EGL stack, but without a nvidia DRM KMS driver, EGL wouldn't be able to use EGLOutput to scanout. Looking at https://github.com/FreeBSDDesktop/kms-drm, it might actually be relatively easy to grab nvidia-drm.ko source code from the Linux .run package (as @aritger also suggested) and use it as-is (or with very few modifications) using the linux-kpi compatibility interface. |
|
Thank you guys for pointing me in the right direction! I'd certainly be interested in giving it a try. I'll have to take a look at the linux nvidia-drm.ko and see if it can be plugged into the linuxkpi. Is all of nvidia-drm open source? If so I'd rather just port it to FreeBSD than rely on the linuxkpi. I'd need to read up on the FreeBSD DRM interface before either of these though. Another quick question, is Vulkan supported? In the 390.xx drivers libEGL had the vkGetProcAddress symbol for the vulkan loader, but it seems to be removed in 415.25. Thanks! |
Back at my desktop so I can answer part of this for myself, I do see all the sources in the nvidia-drm folder. Obviously the nvidia-drm-linux.c file is very linux specific, but how about the rest? some of the others like nvidia-dma-fence-helper.c seem to be linux specific as well but most don't appear to be. How closely tied are the remaining files? (i.e. is it a feasible port or would it involve a total rewrite?) Thanks again for letting me pester you with questions! |
|
@ashaferian nvidia-drm sources are linux-centric overall. I don't think we have been very good at keeping linux-specific bits contained in nvidia-drm-linux.c, sorry. I'm not a FreeBSD expert myself, but it seems the DRM APIs in FreeBSD and Linux differ considerably, so a port might entail more work than the linuxkpi approach. As for your Vulkan question, we don't support Vulkan on FreeBSD. There have been changes related to GLVND, and that's probably the cause for vkGetProcAddress not being exported by libEGL, since now it's GLVND's. |
|
@mvicomoya Now that I think about it I recall seeing a mailing list post about how the linuxkpi based drm was what they were going to use going forward. I think the old drm you mentioned is getting removed next major release. I'll have to double check that but Linuxkpi seems the way to go. So there wouldn't be any problems having nvidia-drm using the linuxkpi while nvidia.ko runs natively on FreeBSD? It sounds fine in theory but I figured I'd double check before I start investigating. Ah that would make sense. Explains the removal of the symbol. Thanks! |
Porting nvidia-drm to FreeBSDApologies for the long message @mvicomoya @aritger Thanks for your advice from this thread, I have thrown some time into porting the nvidia-drm module using FreeBSD's linuxkpi and have some questions about a roadblock I have experienced. I'm sure I have made mistakes so please let me know if there is anything I can correct. I placed what I have so far in a repo so that I can get feedback (and hopefully others can test). From reading Nvidia's license it seems like this was okay to do based off the linux/BSD exception. If this is not allowed, or is a problem in any way please let me know so that I can remove it. Current Statusnvidia-drm.ko builds and can be loaded on FreeBSD using the linuxkpi. /dev/dri/card0 is created and suspend/resume is not broken. This repo (eglstreams-kms-example) does not work however, as it 1) appears to need to link with libglvnd and 2) cannot perform drmModeGetResources. All my work was tested on a GTX 1070. A patch for pci id detection had to be added to the linuxkpi, so this port has to be run on FreeBSD 13.0-CURRENT revision 343451 or later. Temporary hacks which need to be resolved at some point
The current problem (why I am writing to you)One of the conftest options (NV_DRM_ATOMIC_MODESET_AVAILABLE) seems to be what actually enables most of the drm functionality. If this option is not enabled, nvidia-drm can be loaded successfully. It seems to work until the drmModeGetResources call, where it fails because this modeset option is not enabled. I also had to implement some of the missing nvkms API functions in nvidia-modeset.ko. They appear to have been left out as nvidia-modeset originally didn't need them due to a lack of drm support. The actual point of failure is line 303 of nvidia-drm-drv.c returning
All in all there this port has some problems, and is far from perfect. It wasn't bad getting the open source portions ported, but now that the closed source allocation call is failing I could use a nudge in the right direction. I am still interested in getting this working so please tell me if there is anything I can do to resolve these issues and get this port functional. I can also provide a more in depth write up of what I actually had to change to get to this point, if it would be useful for future work on the FreeBSD driver. Thank you so much for your time, I really appreciate your help. Austin Shafer |
|
Nice work, Austin :) It is definitely fine to host your work in progess on github. For the foreseeable future, I think we want to distribute the official nvidia-drm source as part of the NVIDIA driver package (we don't make guarantees about source or binary compatibility between NVIDIA driver components from different releases). So, yes, we'll need to find some compromise with the FreeBSD drm developers for access to drm header files. libglvnd is a vendor-neutral dispatch layer for OpenGL, GLX, and EGL: https://github.com/NVIDIA/libglvnd AFAIK, it is enabled on FreeBSD. What specifically is the problem you're seeing with trying to use libglvnd? Yes, NV_DRM_ATOMIC_MODESET_AVAILABLE is needed for nvidia-drm to support DRM Kernel Modesetting (KMS). I'm sorry the code for nvKms->allocateDevice() is not currently publicly available. Reading through it, it will call back to nvidia-modeset-freebsd.c for: (and possibly others I missed) It looks like you already have tracing code in nvidia-modeset-freebsd.c. Are any of those calls failing? For your nvkms_open_from_kapi() question, I think the important part is that it calls nvKmsOpen(), which it looks like your nvidia-modeset-freebsd.c does. For background:
My guess at this point is that nvidia-modeset-freebsd.c is missing something that the NVKMS "kapi" layer is expecting. Try to trace all the nvidia-modeset-freebsd.c functions and see if any of those are failing. If that still doesn't help shed light on things, then a next step could be to run nvidia-drm.ko on Linux, and trace the functions in nvidia-modeset-linux.c. I would expect the flow on nvidia-modeset-linux.c and nvidia-modeset-freebsd.c to match. Maybe that will help make it clearer what is different. Sorry, that debugging may be a big tedious. Good luck! |
|
Thanks for your reply :)
I assume they don't install the headers because no one else uses them, so I think If I get this ported they should be open to having their port install them.
No problems, I just wanted to make sure that was the proper way to get to EGL now. This repo wants to link with libEGL directly and fails, I wanted to make sure glvnd was the way to go before I spent time making a new minimal test app.
Hmm I don't remember seeing or implementing this one. I'll have to add it. I don't think I get far enough for this to be the root cause, but its impossible to tell until I implement it. Do I need to do anything special to register these functions I'm adding to the kapi? I wonder if I missed a step and thats why my open function isn't being called.
The only call that I see in For reference here is the debugging output from Its far from the greatest debugging output, but I can see the return from nvkms_sema_alloc (which should mean success). From reading the disassembled nvKms->allocateDevice It appears the next call is the
So should I use the
I'll try adding print statements to every kapi call and see if I'm missing one. I only added statements to ones I added so I think your hunch is right and something is failing silently.
Thanks! I suspect I'll have to pass my gpu through to a VM or something so I can step through it live. With my limited experience with this code its kind of hard to just look at the methods and tell whats going wrong. Thanks again for all the help! |
|
If nvkms_sema_alloc() returns non-NULL, then the next thing nvKms->allocateDevice() will do is call nvkms_open_gpu(). Can you add some tracing to it? nvkms_call_rm() is already implemented in nvidia-modeset-freebsd.c. It is just that you may want to add tracing to it. But, I suppose its data is opaque, so there isn't much to be learned by tracing it. For nvkms_open_from_kapi(), I think it will be best to pattern after the nvidia-modeset-linux.c version, which just calls nvkms_open_common() |
|
Yup and nvkms_call_rm is called super frequently, so it just seemed like noise when I traced it. Made some progress, turns out I was only tracing the nvkms_open_from_kapi, however Cool I'll stick with the current linux modeled implementation of nvkms_open_common unless I run into problems with it. Thanks! |
If you don't mind random strangers interjecting, is there currently any plan to support Vulkan on FreeBSD in the future? |
|
We don't have current plans for Vulkan support on FreeBSD. But, knowing it is important to users helps us prioritize future work. |
|
Admittedly, I've only seen about a dozen or so complaints (like this one), which is not much considering it's been 3 years. What really annoys me is zero communication regarding supported features in Linux/Solaris/FreeBSD driver — even a simple comparison table posted somewhere on devtalk.nvidia.com would be nice. |
|
Hi @aritger, Back with some more questions, One is a bug I think I've stumbled upon and the other is about gem page faulting. Also would like to second adding vulkan support. I'd much rather be interacting with Vulkan than EGL, as Vulkan definitely seems to be the future. Out of curiosity is Vulkan dependent on nvidia-drm? (in the linux driver at least) Nvidia-modeset Acquiring Duplicate LocksKernel Panic: nvidia-modeset tries to acquire os.lock_sx @ nvidia_os.c:651 despite currently holding the lock Steps to replicate - Just run Can be encountered on a clean version of 415.25. It's pretty trivial to replicate, although the result can be different depending on how you trigger it. Behavior can range from warnings/kernel panic during module loading to crashing the system later while trying to start x11. This is in the closed source portion of the driver so there's nothing I can do. It'd be great if you have a FreeBSD CURRENT test system that you can install 415.25 on see. While debugging I enabled most of the kernel debugging options available for FreeBSD. The strict lock checking from the witness debugger notices locks being acquired while they are already held. I am on 12-RELEASE, but FreeBSD 13-CURRENT should have the same results, with more strict checking. Normally on a release branch this isn't a problem at all, it just gets ignored. However the CURRENT branch of FreeBSD (the development branch) always has the strict lock checking enabled. It's my guess that the port maintainer for the nvidia driver encountered this on their development system, assumed it was also broken on 12.0, and won't pull 415+ until it works on CURRENT. Just for reference, here are the exact debug options I added to my kernel config (/usr/src/sys/amd64/conf/GENERIC): Please let me know if there is anything you need me to test or do for this. How does __nv_drm_vma_fault work? (This is my real problem)Update - Porting nvidia-drm is going relatively well, I can load the module, open /dev/dri/card0, and generally get pretty far before things start going wrong. Thanks again for all your advice along the way! Problem - I can't seem to process the GEM nvkms page faults correctly. I don't find the correct I have read up on lwn.net about GEM and fault handling in Linux, and I found this which appears to be the Intel equivalent from the FreeBSD linuxkpi-based drm tree. The relevant code is here
Just for reference, and to others reading this, here is my (broken) FreeBSD fault handling equivalent: If any FreeBSD people see this and know what I've done wrong please feel free to comment. My work is stored in the git repo linked earlier. As always please let me know if there is anything I can do differently to get this to work. |
No, Vulkan doesn't depend on nvidia-drm. At least not the basic functionality, since it actually works under Linuxulator. |
|
Oh cool, I assume thats what your repository is for right? When you say it works is that with or without x11 running? |
Not quite, my repo contains a crude glibc shim allowing loading Linux libGL.so (& friends) into a regular FreeBSD process without Linuxulator.
With X11. |
|
Hi Austin. Yes, I've taken note of the FreeBSD Vulkan requests. nvidia-drm is not required for Vulkan, either within X11 or for VK_KHR_display. I've filed NVIDIA internal bug 2507077 for the duplicate lock issue (I don't believe you can access the bug report, but you can at least use that number as a handle if asking about it in the future). Virtual memory management is not my forte, but here is my understanding (looking at the Linux version of nvidia-drm-gen-nvkms-memory.c):
Hopefully that helps with some context. What sorts of values do you get for page_offset if you compute it like this? are those small values, less than PAGE_SIZE? Looking at the FreeBSD linuxkpi-based drm tree you pointed to, does just this give you something sane? It might also help to sprinkle some printouts in the Linux version of __nv_drm_vma_fault() to get a better understanding of the values that it computes. I hope that helps. |
|
Hi @aritger Thanks! I'm also happy to test anything regarding that bug if needed. Also thanks for the explanation, its very helpful. I'll take a look at finding the correct physical location this weekend, but I'll follow up with a few things first.
so
I get something like
This does, although the vm_page_t is busy, I assume this is because it was already mapped/in use from nvidia.ko. I'm still learning the ins and outs of FreeBSD vm so I'm exploring around to see what is and isn't normal. I can also get valid vm_page_t structs (that may or may not be busy) with the page_offset stuff that is currently in my code. Thanks! |
|
@ashaferian, if you haven't seen it, this looks like a helpful resource for what you're investigating: I'm not too familiar with drm_vma_node_start(), but hopefully the above document helps. |
|
Thanks! I had actually read the first part about the concepts and then got sidetracked by trying to fix things. Looking at it again made me notice the function descriptions, including |
|
Hi @aritger I've made some progress, and have run into an issue with EGL (that I assume is coming from a drm initialization problem). I don't have source for libEGL_nvidia so I was hoping you could point me in the right direction. Where I am atI solved the page faulting problem from my previous messages. (For anyone reading this in the future, it came from me not properly setting the vma area's pfn_first and pfn_count fields.) The repo I linked earlier has the changes pushed, the commit messages have more details. Running this eglstream example repo I can make it all the way through drm setup and mode setting! (kms.c) The screen goes black as the vt sets itself to graphical mode, but EGL is not initialized (egl.c) so there is no data to display Current ProblemHere's the bit in egl.c where it breaks:
I've tried dtracing around as its been really helpful with the kernel part of this port, but I can't seem to get anywhere without the source. There are no calls to the EGL linking questionWhats the proper way (simplest case) to link with EGL using the 415.25 driver? I see that there is a libEGL.so in the glvnd folder, and a libgEGL_nvidia.so seems to be installed. The 410* driver series is currently being brought into the FreeBSD port tree (see bugzilla ), so glvnd isn't yet available through a package manager. I assume Nvidia has some docs or tutorials on getting started with glvnd, but I can't seem to find anything about a simple EGL example. (Also I assume glvnd is used and not linking with libEGL directly) Thank you so much! |
|
@ashaferian, nice debugging. I stepped through things on the EGL side on Linux, and it looks like there is a good chunk of the code that interfaces EGL and DRM that is only built into the Linux EGL binary, but not the FreeBSD EGL binary. Sigh. Sorry :( I think that explains why your output layer query is returning n==0. I'll see what I can do to correct the FreeBSD build of EGL for a future NVIDIA driver release. In the meantime: is running the Linux EGL driver through the linuxulator viable to help test your nvidia-drm implementation? Other DRM-KMS tests might include the xf86-video-modesetting X11 driver. For EGL linking, there is some stale, Linux-centric documentation about packaging glvnd here: The intent is that libEGL.so should be provided by glvnd, and that NVIDIA just provides a libEGL_nvidia.so that gets loaded by glvnd. On Linux, we bundle a glvnd libEGL with the driver .run file, and at install time detect if the distro has a distro-provided glvnd libEGL; if not, we install the one in the driver package. I forget what we do on FreeBSD, but I suspect we unconditionally install the driver-packaged glvnd libEGL.so. In any case, applications should just be able to link against libEGL.so, independent of any particular driver implementation. |
|
@aritger I'm sorry to hear that :( If you manage to get it included at some point please let me know! Would introducing the DRM changes to FreeBSD's EGL mess with EGL users that aren't using DRM? I can't seem to get the linuxlator working with EGL, although I think it's within the linuxlators ability. I had some trouble linking with the libraries from the Nvidia driver because they are 32-bit, and I need to go track down a libdrm package for things to work. I can try this at some point in the future when I have a little more free time. Also thanks for the link, Its definitely helpful even if it's a little out dated. Maybe I'll help the committers finally get glvnd included in ports. And from a quick look I think you are right, and Nvidia just automatically makes a link for libEGL.so to the libEGL_nvidia. I vaguely remember hearing this causes problems if you install mesa-libs too, but I could be wrong. Somewhat unrelated, but even though the EGL problems spell doom for this port, I did have a lot of fun learning how to do it. I'm looking for an internship before I go to grad school this fall, and if you all are still hiring interns for this summer I'd love to keep working on stuff like this. Thank you so much for all of your help! |
I've got a suspicion you might be interested in looking at CUDA stuff. Specifically a few stubs in nvidia.ko: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=224358#c15.
If you have installed nvidia-driver port, you should also have 64-bit Linux libraries.
Not really, libEGL.so is one of the glvnd wrapper libraries. |
I have a quick question on some EGL extensions for the FreeBSD driver that can’t seem to be found. I’m posting here because the Nvidia forums for FreeBSD don’t seem to get any answers from developers and I found this issue trying to run this repository. If there is a better place for me to ask this question please let me know.
I am trying to write a small wayland compositor on FreeBSD, and while testing the eglstream example the FreeBSD driver reports that It can’t find certain extensions that patchnotes say are included.
Replication on FreeBSD 12.0-RELEASE amd64 on a GTX 1070 with nvidia-driver 390.87_1
(pkg install gmake …)
git clone https://github.com/aritger/eglstreams-kms-example
cd eglstreams-kms-example
gmake <——— Compilation with gmake works fine
running ./eglstreams-kms-example results in:
ERROR: eglGetProcAddress(eglGetOutputLayersEXT) failed
Some reading found that the extensions are searched for in the “utils.c” file. The 346 driver notes say:
Added support for the following EGL extensions:
EGL_EXT_device_base
EGL_EXT_platform_device
EGL_EXT_output_base <— It looks like this is the one missing
I don’t see EGL_EXT_output_base in the nvidia-settings app either. Is this repository a good way to get an OpenGL context without X11 on FreeBSD? And are these extensions still supported?
Please let me know if there is any more information I can provide, I am more than happy to help in any way to get this working, as it is a feature I am extremely interested in.
Thank you so much for your time!
Austin Shafer
The text was updated successfully, but these errors were encountered: