different errors on **libpython-clj.python/initialize!** between 1.46 and 2.00-alpha-5

[temco]

PC environment：
macOS Catalina 10.15.7

emacs environment:

python environment:
installed by conda (v3.8.3)
executable: /opt/anaconda3/bin/python3; packages: /opt/anaconda3/lib/python3.8/site-packages/

installed by commandLineTools of Xcode (v3.7.3)
executable: /Library/Developer/CommandLineTools/usr/bin/python3; packages: ~/Library/Python/3.7/lib/python/site-packages/

Before yesterday, my python3 command in the terminal was linked by conda, also v3.7.3. In my clojure code that called the python functions with libpython-clj "1.46", I initialized like

(initialize! :python-home "/opt/anaconda3")

and libpython-clj could find the way to run. When I updated the python's version with conda from 3.7.3 to 3.8.3, and modified the project.clj by converting the version of libpython-clj from "1.46" to "2.00-alpha-5", a series of errors were found.

If I used libpython-clj "2.00-alpha-5" and (initialize! :python-home "/opt/anaconda3"), it said:

clojure.lang.Reflector invokeMatchingMethod
Trying python library names ["/opt/anaconda3/lib" "/opt/anaconda3/lib" "python3.8" "python3.7m" "python3.6m"]
clojure.lang.Reflector invokeMatchingMethod
Setting java library path: /opt/anaconda3/lib:/Users/xxx/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
clojure.lang.Reflector invokeMatchingMethod
Reference thread starting
clojure.lang.Reflector invokeMatchingMethod
Library python3.8 found at [:java-library-path "/opt/anaconda3/lib/libpython3.8.dylib"]
Syntax error (IllegalStateException) compiling at (libpython.clj:8:1).
Failed to find constructor f java.nio.DirectByteBuffer: $s

If I used libpython-clj "1.46" and (initialize! :python-home "/opt/anaconda3"), it said:

clojure.tools.logging$eval8937$fn__8940 invoke
Trying python library names ["/opt/anaconda3/lib" "/opt/anaconda3/lib" "python3.8" "python3.7m" "python3.6m"]
clojure.tools.logging$eval8937$fn__8940 invoke
Setting java library path: /opt/anaconda3/lib:/Users/xxx/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
clojure.tools.logging$eval8937$fn__8940 invoke
Reference thread starting
clojure.tools.logging$eval8937$fn__8940 invoke
Library python3.8 found at [:java-library-path "/opt/anaconda3/lib/libpython3.8.dylib"]
clojure.tools.logging$eval8937$fn__8940 invoke
Library c found at [:system "c"]
Testing xxx.xxx.xxx
*** Closed on Thu Oct 29 16:52:29 2020 ***

It seemed the executable and lib were right but when I require-python, it collapsed.

If I used libpython-clj "2.00-alpha-5" and (initialize! :python-executable "/Library/Developer/CommandLineTools/usr/bin/python3"), it said:

clojure.lang.Reflector invokeMatchingMethod
Trying python library names ["/Users/xxx/Library/Python/3.7/lib/" "python3.7" "python3.7m" "python3.6m"]
clojure.lang.Reflector invokeMatchingMethod
Setting java library path: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib:/Users/xxx/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
clojure.lang.Reflector invokeMatchingMethod
Reference thread starting
clojure.lang.Reflector invokeMatchingMethod
Library python3.7 found at [:java-library-path "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/libpython3.7.dylib"]
Syntax error (IllegalStateException) compiling at (libpython.clj:8:1).
Failed to find constructor f java.nio.DirectByteBuffer: $s
ERROR: Unhandled REPL handler exception processing message {:nrepl.middleware.print/stream? 1, :nrepl.middleware.print/print cider.nrepl.pprint/pprint, :nrepl.middleware.print/quota 1048576, :nrepl.middleware.print/buffer-size 4096, :nrepl.middleware.print/options {:right-margin 70}, :op stacktrace, :session a0a9fd19-6996-4077-a1a9-8d785e7da744, :id 17}
java.lang.NoClassDefFoundError: Could not initialize class xerial.larray.buffer.UnsafeUtil

the same as libpython-clj "2.00-alpha-5" and (initialize! :python-home "/opt/anaconda3")

If I used libpython-clj "1.46" and (initialize! :python-home "/Users/xxx/Library/Python/3.7"), it said:

clojure.tools.logging$eval8937$fn__8940 invoke
Trying python library names ["python3.7m" "python3.7" "python3.7m" "python3.6m"]
clojure.tools.logging$eval8937$fn__8940 invoke
Setting java library path: /Users/xxx/Library/Python/3.7/lib:/Users/xxx/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
clojure.tools.logging$eval8937$fn__8940 invoke
Reference thread starting
Syntax error (ExceptionInfo) compiling at (yyy.clj:8:1).
Failed to load library

and cider-error said (I don't know why it was python3.6):

1. Caused by clojure.lang.ExceptionInfo
   Failed to load library
   {:libname "python3.6m",
    :paths
    [[:system ["python3.6m"]]
     [:java-library-path
      ["/Users/xxx/Library/Python/3.7/lib/libpython3.6m.dylib"
       "/Users/xxx/Library/Java/Extensions/libpython3.6m.dylib"
       "/Library/Java/Extensions/libpython3.6m.dylib"
       "/Network/Library/Java/Extensions/libpython3.6m.dylib"
       "/System/Library/Java/Extensions/libpython3.6m.dylib"
       "/usr/lib/java/libpython3.6m.dylib"
       "./libpython3.6m.dylib"]]]}

If I used libpython-clj "1.46" and (initialize! :python-executable "/Library/Developer/CommandLineTools/usr/bin/python3"), it worked.
However, I only could use libpython-clj "1.46" and python3.7 in this way. I cannot use "2.00-alpha-5", with either python3.8 (by conda) or python3.7 (by commandLineTools), and cannot use python3.8, with either "1.46" or "2.00-alpha-5".

[cnuernber]

Hmm, there are a range of issues there.

First, they removed a constructor for direct nio buffers in java 14. Previously the lower level layers were using jna for that conversion and that apparently still works in java 14. That is this error:

Failed to find constructor f java.nio.DirectByteBuffer: $s

If you don't have things explicitly dependent upon java 14 then I believe java 8 will work.

Before I was using JNA for that pathway (long pointer -> nio buffer) but I removed it because JNA has issues with graal native and nio buffers don't support memory spaces larger than 2G.

Java-14+ has a native memory abstraction that I haven't put time into because the majority of my users are Java-8.

So, I am not sure the best pathway here; I am disappointed it didn't work for you however. Is Java-8 a possibility?

[cnuernber]

Hmm, potentially some of the fix could come from re-enabling unsafe for the jdk:
https://blogs.oracle.com/javamagazine/the-unsafe-class-unsafe-at-any-speed

[temco]

Hmm, there are a range of issues there.

First, they removed a constructor for direct nio buffers in java 14. Previously the lower level layers were using jna for that conversion and that apparently still works in java 14. That is this error:

Failed to find constructor f java.nio.DirectByteBuffer: $s

If you don't have things explicitly dependent upon java 14 then I believe java 8 will work.

Before I was using JNA for that pathway (long pointer -> nio buffer) but I removed it because JNA has issues with graal native and nio buffers don't support memory spaces larger than 2G.

Java-14+ has a native memory abstraction that I haven't put time into because the majority of my users are Java-8.

So, I am not sure the best pathway here; I am disappointed it didn't work for you however. Is Java-8 a possibility?

thanks, I have removed the java 14, so that "2.00-alpha" and python 3.7 work together.

[jjtolton]

Thanks for the thoughtful write up @temco !

[cnuernber]

I think we should leave this issue up. At some point we can address this with jdk-14 specific pathways.

[cnuernber]

@temco - JDK-14 should work out of the box with 2.00+. Let me know if this is still an issue but I did take a pass and I fixed all the obvious issues with our toolchain w/r/t jdk-14.