Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Produce usable / distributable macosx wheels #79

Closed
devinrsmith opened this issue Aug 19, 2022 · 0 comments
Closed

Produce usable / distributable macosx wheels #79

devinrsmith opened this issue Aug 19, 2022 · 0 comments

Comments

@devinrsmith
Copy link
Member

Currently, the macosx wheels are being built with a hard-coded CI paths (/Users/runner/hostedtoolcache/Java_Temurin-Hotspot_jdk/8.0.332-9/x64/Contents/Home/jre/lib/server). This makes them unfit for distribution.

$ delocate-listdeps jpy-0.11.0-cp38-cp38-macosx_10_15_x86_64.whl      

@rpath/libjvm.dylib not found:
  Needed by: /private/var/folders/7z/khph2v4j2pzdf6rmc9gljvww0000gn/T/tmp3s8etg8z/jpy.cpython-38-darwin.so
  Search path:
    /Users/runner/hostedtoolcache/Java_Temurin-Hotspot_jdk/8.0.332-9/x64/Contents/Home/jre/lib/server
...
$ otool -l jpy.cpython-38-darwin.so

...
Load command 10
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name @rpath/libjvm.dylib (offset 24)
   time stamp 2 Wed Dec 31 16:00:02 1969
      current version 1.0.0
compatibility version 1.0.0
...
Load command 12
          cmd LC_RPATH
      cmdsize 112
         path /Users/runner/hostedtoolcache/Java_Temurin-Hotspot_jdk/8.0.332-9/x64/Contents/Home/jre/lib/server (offset 12)
...

Unfortunately, just removing the hard-coded LC_RPATH is not enough to produce usable wheels; @rpath/libjvm.dylib is not resolvable.

Going down a rabbit-hole, I discovered that dynamic loading / dlopen semantics are different on Linux vs OS X.

On Linux, once a library has been opened, it can be resolved via its soname:

>>> import ctypes

>>> ctypes.CDLL("libjvm.so")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libjvm.so: cannot open shared object file: No such file or directory

>>> ctypes.CDLL("/usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so")
<CDLL '/usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so', handle 55993695f530 at 0x7f55e89e3010>

>>> ctypes.CDLL("libjvm.so")
<CDLL 'libjvm.so', handle 55993695f530 at 0x7f55e89e32b0>

The same is not true on Mac:

>>> import ctypes

>>> ctypes.CDLL("libjvm.dylib")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/homebrew/Cellar/python@3.10/3.10.6_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(libjvm.dylib, 0x0006): tried: '/opt/homebrew/lib/libjvm.dylib' (no such file), 'libjvm.dylib' (no such file), '/usr/local/lib/libjvm.dylib' (no such file), '/usr/lib/libjvm.dylib' (no such file), '/Users/deephaven/libjvm.dylib' (no such file)

>>> ctypes.CDLL("/opt/homebrew/Cellar/openjdk@11/11.0.16.1/libexec/openjdk.jdk/Contents/Home/lib/server/libjvm.dylib")
<CDLL '/opt/homebrew/Cellar/openjdk@11/11.0.16.1/libexec/openjdk.jdk/Contents/Home/lib/server/libjvm.dylib', handle 2013663c0 at 0x100ee0100>

>>> ctypes.CDLL("libjvm.dylib")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/homebrew/Cellar/python@3.10/3.10.6_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(libjvm.dylib, 0x0006): tried: '/opt/homebrew/lib/libjvm.dylib' (no such file), 'libjvm.dylib' (no such file), '/usr/local/lib/libjvm.dylib' (no such file), '/usr/lib/libjvm.dylib' (no such file), '/Users/deephaven/libjvm.dylib' (no such file)

https://developer.apple.com/forums/thread/132817 seems to reinforce that fact.

Note: if we were re-distributing a JVM we'd be able to setup a proper LC_RPATH / @rpath, but we're wanting the user to bring-their-own JVM (either as resolvable at runtime via the PATH or via JAVA_HOME).

Ultimately, I think the solution, currently in testing, is to remove both LC_RPATH and the LC_LOAD_DYLIB for libjvm.dylib; we'll have appropriate bootstrapping application logic to load libjvm.dylib, and the .so files will not try and re-find an rpath'd libjvm.dylib.

devinrsmith added a commit to devinrsmith/jpy that referenced this issue Aug 21, 2022
This has the effect of removing the non-portable LC_LOAD_DYLIB and LC_RPATH directives for libjvm.dylib from the .so file.

Fixes jpy-consortium#79, see for more details
devinrsmith added a commit to devinrsmith/jpy that referenced this issue Aug 21, 2022
This has the effect of removing the non-portable LC_LOAD_DYLIB and LC_RPATH directives for libjvm.dylib from the .so file.

Fixes jpy-consortium#79, see for more details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant