Frozen ctypes.util.find_library("OpenGL") returns None on macOS Big Sur

[papr]

Description of the issue

• On macOS Big Sur, ctypes.util.find_library("OpenGL") returns None when frozen
• On macOS Big Sur, ctypes.util.find_library("OpenGL") returns /System/Library/Frameworks/OpenGL.framework/OpenGL when not froozen
• On earlier versions of macOS, it returns /System/Library/Frameworks/OpenGL.framework/OpenGL (frozen and not frozen equally)

I expect ctypes.util.find_library("OpenGL") to always return the same path, independent of being frozen or not.

This issue might affect other system libraries as well.

See more motivation at the bottom as to why this is relevant.

Context information

Details

* Output of `pyinstaller --version`: `4.2` and `5.0.dev0` * Version of Python: 3.9.1 (via [`macOS 64-bit universal2 installer`](https://www.python.org/downloads/release/python-391/)) * Freezing Platform: `macOS 10.14.6 Mojave` * Runtime Platform: - incorrect behavior on `macOS 11.1 Big Sur` - correct behavior on `macOS 10.15.7 Catalina`

Checklist

• start with clean installation
• use the latest development version
• Run your frozen program from a command window (shell) — instead of double-clicking on it
• Package your program in --onedir mode
• Package without UPX, say: use the option --noupx or set upx=False in your .spec-file
• Repackage you application in verbose/debug mode. For this, pass the option --debug to pyi-makespec or pyinstaller or use EXE(..., debug=1, ...) in your .spec file.

Passing --debug all causes the binary to seg fault on the runtime platform.

A minimal example program which shows the error

Details

Program to be frozen: `find_opengl.py` ```py import ctypes.util

print(ctypes.util.find_library("OpenGL"))

Freeze command:
```sh
pyinstaller \
    -y \
    --clean \
    --log-level ERROR \
    --distpath "dist-$(sw_vers -productVersion)-$(pyinstaller -v)" \
    --onedir \
    --noupx \
    find_opengl.py

Again, passing --debug all causes the binary to seg fault on the runtime platform (Big Sur).

Output

Details

#### macOS 10.15.7 Catalina (frozen on macOS 10.14.6)

$ python find_opengl.py
/System/Library/Frameworks/OpenGL.framework/OpenG

$ dist-10.14.6-5.0.dev0/find_opengl/find_opengl
/System/Library/Frameworks/OpenGL.framework/OpenGL

macOS 11.1 Big Sur (frozen on macOS 10.14.6)

$ python find_opengl.py
/System/Library/Frameworks/OpenGL.framework/OpenGL

$ dist-10.14.6-5.0.dev0/find_opengl/find_opengl
None

macOS 11.1 Big Sur (frozen on macOS 11.1)

$ python find_opengl.py
/System/Library/Frameworks/OpenGL.framework/OpenGL

$ dist-11.1-5.0.dev0/find_opengl/find_opengl
None

Motivation

Details

I am trying to freeze a larger program that depends on [PyOpenGL](https://pypi.org/project/PyOpenGL/). Unfortunately, PyOpenGL is not hosted such that one can directly link to the source. Instead, I am quoting the relevant function:

OpenGL/platform/ctypesloader.py:17-36, called from OpenGL/platform/darwin.py:35

def _loadLibraryWindows(dllType, name, mode):
    """Load a given library for Windows systems

    returns the ctypes C-module object
    """
    fullName = None
    try:
        fullName = util.find_library( name )
        if fullName is not None:
            name = fullName
        elif os.path.isfile( os.path.join( DLL_DIRECTORY, name + '.dll' )):
            name = os.path.join( DLL_DIRECTORY, name + '.dll' )
    except Exception as err:
        _log.info( '''Failed on util.find_library( %r ): %s''', name, err )
        # Should the call fail, we just try to load the base filename...
        pass
    try:
        return dllType( name, mode )
    except Exception as err:
        err.args += (name,fullName)
        raise

• dllType is of type __main__.PyInstallerCDLL
• name is OpenGL
• mode is ctypes.RTLD_GLOBAL
• fullName = util.find_library( name ) returns None on Big Sur
• dllType( name, mode ) causes Pyinstaller can't find dynamically linked libs for Mac OS X Big Sur Beta  #5107 (comment)

OpenGL/platform/darwin.py:32-41

    @baseplatform.lazy_property
    def GL(self):
        try:
            return ctypesloader.loadLibrary(
                ctypes.cdll,
                'OpenGL', 
                mode=ctypes.RTLD_GLOBAL 
            )
        except OSError as err:
            raise ImportError("Unable to load OpenGL library", *err.args)

[bwoodsend]

On macOS Big Sur, ctypes.util.find_library("OpenGL") returns /System/Library/Frameworks/OpenGL.framework/OpenGL when not froozen

I'm surprised by this. Does that file really exist? System libraries are supposed to be hidden on Big Sur. Anything using ctypes.util.find_library() should just use ctypes.CDLL() instead and wrap it in a try statement if you don't expect it to work.

[rokm]

It actually does work as described. But only if 3.9.1 macOS 64-bit universal2 installer is used (if macOS 64-bit Intel installer is used, it returns None, as in older versions).

[papr]

Interestingly, this fails on macOS 10.14, macOS 10.15, and macOS 11.1 (not frozen):

In [1]: import ctypes
   ...: ctypes.CDLL("OpenGL", ctypes.RTLD_GLOBAL)
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-3-7fa9d8669c3d> in <module>
      1 import ctypes
----> 2 ctypes.CDLL("OpenGL", ctypes.RTLD_GLOBAL)

~/.pyenv/versions/3.8.2/Python.framework/Versions/3.8/lib/python3.8/ctypes/__init__.py in __init__(self, name, mode, handle, use_errno, use_last_error, winmode)
    371
    372         if handle is None:
--> 373             self._handle = _dlopen(self._name, mode)
    374         else:
    375             self._handle = handle

OSError: dlopen(OpenGL, 10): image not found

This is probably why PyOpenGL passes the result of ctypes.util.find_library("OpenGL") to ctypes.CDLL explicitly, because there does not seem to be an other way than that.

[rokm]

Aha, figured it out... since our bootloader is not built for Big Sur (i.e., the SDK we build against is probably older than 11.0), the platform identification is using compatibility 10.16 identifier instead of 11.x one.

And this makes _ctypes._dyld_shared_cache_contains_path() unavailable at runtime.

E.g.,

# find_c_library.py
import platform
import ctypes.util
print("Platform:", platform.mac_ver())
print(ctypes.util.find_library("c"))

Unfrozen script:

$ python find_c_library.py 
Platform: ('11.2', ('', '', ''), 'x86_64')
/usr/lib/libc.dylib

Frozen script:

$ ./dist/find_c_library/find_c_library 
Platform: ('10.16', ('', '', ''), 'x86_64')
None

Frozen script, with version compatibility mode disabled:

$ SYSTEM_VERSION_COMPAT=0 ./dist/find_c_library/find_c_library 
Platform: ('11.2', ('', '', ''), 'x86_64')
/usr/lib/libc.dylib

[bwoodsend]

I take it then that this would be another fat binary like in #5315? i.e. Build with --mmacos-min-version=10.7, rebuild with --mmacos-min-version=11.0, then lipo the two together. This seems to be a very inefficient way to maintain cross compatibility.

[rokm]

I take it then that this would be another fat binary like in #5315?

I don't think so - I think we'd just need to build against 11.0 SDK, but we can leave the minimum version at lower version.

For example, if you run otool -l against the python interpreter binary, you get:

Load command 9
      cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.9
      sdk 11.0

(which is consistent with the note on python.org that this version should still run on 10.9 or newer).

In our bootloader's case, we get:

Load command 9
      cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.7
      sdk 10.11

[bwoodsend]

Ah, that's not too bad then. I imagine this can be set somewhere in the vagrant file in the bootloader folder.

[rokm]

We can close this now, as bootloaders in just-released PyInstaller 4.3 have been built against recent MacOS SDK, and so ctypes.util.find_library() should work on Big Sur as well (provided Python version is recent enough).