Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

primus breaks on fedora with mesa-13.0.3-6 #193

Open
gsgatlin opened this Issue Feb 2, 2017 · 33 comments

Comments

Projects
None yet
6 participants
@gsgatlin
Copy link

gsgatlin commented Feb 2, 2017

Hello.

Nvidia has created a new library called libglvnd https://github.com/NVIDIA/libglvnd

The purpose is "This is a work-in-progress implementation of the vendor-neutral dispatch layer for arbitrating OpenGL API calls between multiple vendors on a per-screen basis, as described by Andy Ritger's OpenGL ABI proposal"

Fedora is adding this to all fedora 25 machines in a few days. Mesa has been redone to support this new library.

It looks like they have implemented it in 4 patches to "mesa." I can post the patches if its important?

With these changes "VirtualGL" bridge continues to work but "primus" bridge no longer works.

Here is the output from command line:

https://paste.fedoraproject.org/541783/14858971/

(primusrun also does not work)

Does anyone have any clues on how to get around that? I have opened:

https://bugzilla.redhat.com/show_bug.cgi?id=1418103

Here are some logs: (Can't really see anything in them)
/var/log/messages
first run working with 13.0.3-5
https://paste.fedoraproject.org/541695/58843511/
second run failing with 13.0.3-6
https://paste.fedoraproject.org/541707/14858848/
/var/log/Xorg.8.log
first run working with 13.0.3-5
https://paste.fedoraproject.org/541709/88496314/
second run failing with 13.0.3-6
https://paste.fedoraproject.org/541712/85885065/

I'm sorry I don't have any ideas on how to fix this. I did notice

/usr/lib64/libGL.so.1: undefined symbol: _glapi_tls_Current

in the output on the commanbd line.

Please let me know if I can provide any further information or be of any help with this issue. Thanks.

@jwrdegoede

This comment has been minimized.

Copy link

jwrdegoede commented Feb 2, 2017

The: "/usr/lib64/libGL.so.1: undefined symbol: _glapi_tls_Current" error is weird, what does:

rpm -qf /usr/lib64/libGL.so.1 say ? And does: ldd -r /usr/lib64/libGL.so.1 give the same error ?

On my system with libglvnd-0.2.999-7.gitdc16f8c.fc25 and mesa-libGL-13.0.3-6.fc25.x86_64 this gives:

[hans@shalem master]$ rpm -qf /usr/lib64/libGL.so.1
libglvnd-glx-0.2.999-7.gitdc16f8c.fc26.x86_64
[hans@shalem master]$ ldd -r /usr/lib64/libGL.so.1
linux-vdso.so.1 (0x00007ffc171ba000)
libGLX.so.0 => /lib64/libGLX.so.0 (0x00007f87be9a9000)
libX11.so.6 => /lib64/libX11.so.6 (0x00007f87be66a000)
libXext.so.6 => /lib64/libXext.so.6 (0x00007f87be458000)
libGLdispatch.so.0 => /lib64/libGLdispatch.so.0 (0x00007f87be1a2000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f87bdf9e000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f87bdd7e000)
libc.so.6 => /lib64/libc.so.6 (0x00007f87bd9b8000)
libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f87bd790000)
/lib64/ld-linux-x86-64.so.2 (0x00005558c3bd1000)
libXau.so.6 => /lib64/libXau.so.6 (0x00007f87bd58c000)

Notice that ldd -r does not give the error you are seeing. I've a feeling you've somehow overwritten your libGL.so.1 with one from a different source.

@gsgatlin

This comment has been minimized.

Copy link
Author

gsgatlin commented Feb 2, 2017

Hi. Here is the output from those commands.

[gsgatlin@y470c ~]$ rpm -qf /usr/lib64/libGL.so.1
libglvnd-glx-0.2.999-7.gitdc16f8c.fc25.x86_64
[gsgatlin@y470c ~]$ ldd -r /usr/lib64/libGL.so.1
linux-vdso.so.1 (0x00007fff04dbb000)
libGLX.so.0 => /lib64/libGLX.so.0 (0x00007f481ce61000)
libX11.so.6 => /lib64/libX11.so.6 (0x00007f481cb22000)
libXext.so.6 => /lib64/libXext.so.6 (0x00007f481c910000)
libGLdispatch.so.0 => /lib64/libGLdispatch.so.0 (0x00007f481c65a000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f481c456000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f481c236000)
libc.so.6 => /lib64/libc.so.6 (0x00007f481be70000)
libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f481bc48000)
/lib64/ld-linux-x86-64.so.2 (0x000055b3fb143000)
libXau.so.6 => /lib64/libXau.so.6 (0x00007f481ba44000)

@jwrdegoede

This comment has been minimized.

Copy link

jwrdegoede commented Feb 2, 2017

@gsgatlin

This comment has been minimized.

Copy link
Author

gsgatlin commented Feb 2, 2017

Hello. Here is what the "primusrun" script looks like.

https://paste.fedoraproject.org/543062/14860483/

There is some manipulation of LD_LIBRARY_PATH

@jwrdegoede

This comment has been minimized.

Copy link

jwrdegoede commented Feb 2, 2017

@jwrdegoede

This comment has been minimized.

Copy link

jwrdegoede commented Feb 2, 2017

@gsgatlin

This comment has been minimized.

Copy link
Author

gsgatlin commented Feb 2, 2017

Hello,

could this be what you needed for overview?

https://github.com/amonakov/primus/blob/master/technotes.md

Will try to uncomment the lines and test once I can get back to my office. Cheers.

@jwrdegoede

This comment has been minimized.

Copy link

jwrdegoede commented Feb 2, 2017

@gsgatlin

This comment has been minimized.

Copy link
Author

gsgatlin commented Feb 2, 2017

If I comment the export LD_LIBRARY_PATH line it opens a window but it uses the wrong opengl. (INTEL)

If I comment out PRIMUS_libGL it segfaults.

But thanks a lot for taking a deep look into this. Perhaps some others will have some ideas. Cheers.

@gsgatlin

This comment has been minimized.

@jwrdegoede

This comment has been minimized.

Copy link

jwrdegoede commented Feb 6, 2017

@gsgatlin

This comment has been minimized.

Copy link
Author

gsgatlin commented Feb 6, 2017

Well, I need to test it and see if the primusrun script could be altered to work using his workaround. If it works would need to see if the "optirun" program could also be changed to work somehow. That will be harder since its written in C I think.

@gsgatlin

This comment has been minimized.

Copy link
Author

gsgatlin commented Feb 6, 2017

But I will try to update the wiki one way or the other.

@gsgatlin

This comment has been minimized.

Copy link
Author

gsgatlin commented Feb 6, 2017

Did not seem to work for me.

https://paste.fedoraproject.org/549972/48640003/

Oh well. Still open to any and all suggestions. Do not want to have to switch distros eventually.

@gsgatlin

This comment has been minimized.

Copy link
Author

gsgatlin commented Feb 7, 2017

Alright. I did more experimentation.

The trick is that

__GLVND_DISALLOW_PATCHING=1

needs to be set AND the nvidia shim must be rebuilt/recompiled AFTER libglvnd has been added by mesa. This could be accomplished by running

bumblebee-nvidia --force

by hand as root. Or by waiting until the next kernel upgrade. So that information will need to be added to the wiki.

Things that need to be done.

patch primusrun in the primus rpm to include this variable. (It must be exported)
do a "pull request" to get it added upstream if @amonakov approves of it. I think this change could be beneficial to fedora or arch linux users who have libglvnd.

try to see if this can be set within "optirun" using the setenv() syscall.
If it can and if it works, do another pull request on that change as well.

I will be pretty busy today in meetings but hopefully it can be fixed later in the week.

@gsgatlin

This comment has been minimized.

Copy link
Author

gsgatlin commented Feb 10, 2017

I must have done something wrong in my earlier testing. oh well.

In any event, I have fixed bumblebee and primus rpms. all ready to go. But I also need to update the bumblebee-nvidia driver package. (Both types) because it needs a new argument to the installer to work. The new arg is:

--no-install-libglvnd

So I will work on that this afternoon. One I test it a bit more I will update all three today and update the wiki.

@jwrdegoede do you know if they are planning to add libglvnd to rhel 6 or rhel 7 distros? Thanks.

@gsgatlin

This comment has been minimized.

Copy link
Author

gsgatlin commented Feb 10, 2017

Ok. Pushed new primus, bumblebee, and bumblebee-nvidia rpms to my third party yum repositories.

updated wiki page with:

https://fedoraproject.org/wiki/Bumblebee#mesa-13.0.3-6_and_libglvnd_issues

Will try to do pull requests next week in this project and the bumblebee project in case it can help other bumblebee users on some other distros. Sorry there was not time this week to deal with that.

@jwrdegoede

This comment has been minimized.

Copy link

jwrdegoede commented Feb 11, 2017

@jwrdegoede

This comment has been minimized.

Copy link

jwrdegoede commented Feb 11, 2017

@ArchangeGabriel

This comment has been minimized.

Copy link

ArchangeGabriel commented Feb 12, 2017

@gsgatlin Thanks for your continued great work on Bumblebee stuff. Could you please list modifications you’ve made to your packages to make it work? ArchLinux is going to enable libglvnd too, so it would be nice to have them there too. ;)

@gsgatlin

This comment has been minimized.

Copy link
Author

gsgatlin commented Feb 13, 2017

Hey, sorry for delay @ArchangeGabriel . had to work on training today. Hopefully can work on pull request tomorrow. (If that's appropriate)

so here are some links to some patches and log files.

this patch is for primus:
primus-1.1-libglvndfix.patch

https://paste.fedoraproject.org/557353/14870220/

this patch is for bumblebee:
bumblebee-libglvndfix.patch

https://paste.fedoraproject.org/557355/14870221/

also, for my wrapper for the nvidia-installer program the following changes were needed:
https://paste.fedoraproject.org/557357/14870221/

If I did not at least add "--no-install-libglvnd" it did not work with this error in the log:

https://paste.fedoraproject.org/557358/14870221/

So I added:
"--no-libglx-indirect --no-install-libglvnd --no-glvnd-glx-client --no-glvnd-egl-client"

on machines with libglvnd rpm installed just to be extra paranoid.

So if I were to do pull request once I clean up my account on github.com is there are certain branch I should use? (I seem to recall seeing that somewhere) for primusrun changes and bumblebee changes in optirun.c? Not sure how you will deal with the drivers package on arch since there is an official one for the arch distro unlike fedora which has several nvidia drivers packages including mine which is just for bumblebee.

Thanks,

@gsgatlin

This comment has been minimized.

Copy link
Author

gsgatlin commented Feb 14, 2017

Actually, the regular arch linux nvidia package may just work. I think the environmental variable is the more important piece of it.

@ArchangeGabriel

This comment has been minimized.

Copy link

ArchangeGabriel commented Feb 14, 2017

For Bumblebee, just do a PR against the develop branch. But please change the comment to a more generic one like this: /* primus needs this variable workaround for libglvnd enabled mesa */.

@Lekensteyn

This comment has been minimized.

Copy link

Lekensteyn commented Feb 19, 2017

@SolarAquarion opened a PR at Bumblebee-Project/Bumblebee#845 with the envvar change, but the explanation (commit message and comment) are not clear to me. Can someone have a look?

@kmare

This comment has been minimized.

Copy link

kmare commented Feb 26, 2017

Just a heads up for people still having issues on fedora loading the driver after the libglvnd update. I had to add a selinux policy to make it work (of course after I had ran bumblebee-nvidia --force).

ausearch -c 'modprobe' --raw | audit2allow -M my-modprobe
semodule -X 300 -i my-modprobe.pp

after that, it worked just fine.

@hevanaa

This comment has been minimized.

Copy link

hevanaa commented Feb 26, 2017

I noticed the SELinux problem on Fedora 25, too. But in addition, Steam suddenly crashes with "Fatal Error. Failed to load steamui.so" and adding LD_PRELOAD works:

LD_PRELOAD='/usr/$LIB/libstdc++.so.6 /usr/$LIB/libgcc_s.so.1 /usr/$LIB/libxcb.so.1' primusrun steam

Steam works fine by itself without primusrun.

@gsgatlin

This comment has been minimized.

Copy link
Author

gsgatlin commented Feb 27, 2017

This should be fixed in
bumblebee-nvidia-375.39-2 or bumblebee-nvidia-3.0-3 (un-managed)

It can be dnf updated beginning now.

I updated the selinux security policy so modules should load now. Thanks for the heads up about the problem.

Not sure about steam but I did test "minecraft" and I get 60fps and everything seems to work ok for me.

@kmare

This comment has been minimized.

Copy link

kmare commented Feb 27, 2017

Thank you @gsgatlin , your work is truly appreciated!

@hevanaa

This comment has been minimized.

Copy link

hevanaa commented Feb 28, 2017

Running bumblebee-nvidia-375.39-2 now, but the steam problem still persist. Other applications work fine with primusrun.

@gsgatlin

This comment has been minimized.

Copy link
Author

gsgatlin commented Mar 1, 2017

@hevanaa Do you know if the steam problem is caused by selinux changes or by libglvnd addition? Perhaps you can try "permissive" or "disabled" in /etc/selinux/config after a reboot to see if it affects steam?

@hevanaa

This comment has been minimized.

Copy link

hevanaa commented Mar 1, 2017

Same behavior with SELinux turned off. Steam can't find libraries when started with primusrun. And works fine when started with just "steam".

I tried this:
LD_DEBUG=libs primusrun steam

...
4542: /usr/lib/primus/libGL.so.1: error: relocation error: symbol _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEED1Ev, version GLIBCXX_3.4.21 not defined in file libstdc++.so.6 with link time reference (fatal)
4542: find library=libGL.so.1 [0]; searching
...
4542: trying file=/usr/lib/primus/libGL.so.1
4542:
4542: /usr/lib/primus/libGL.so.1: error: relocation error: symbol _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEED1Ev, version GLIBCXX_3.4.21 not defined in file libstdc++.so.6 with link time reference (fatal)

(here the Fatal error dialog appears)

I can't see anything wrong with libstdc++ on this system.

Sounds like an ABI problem to me.

@gsgatlin

This comment has been minimized.

Copy link
Author

gsgatlin commented Mar 2, 2017

@hevanaa Can you run steam normally but add primusrun to the launch options of some of your higher end games via: https://support.steampowered.com/kb_article.php?ref=6316-GJKC-7437 Or does that fail also?

@hevanaa

This comment has been minimized.

Copy link

hevanaa commented Mar 2, 2017

@gsgatlin Yes, this works fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.