Error requiring libpython-clj2.require on M1 Mac - JDK-18

[Hendekagon]

I'm using libpython-clj 2.018 with aarch64 jdk-18.0.1.jdk, with :jdk-17 alias' :jvm-opts and I get this error when (require '[libpython-clj2.require :refer [require-python]]) on an M1 Mac:

INFO: Unable to find direct buffer constructor - falling back to jdk16 memory model. Reflection warning, tech/v3/datatype/ffi/nio_buf_mmodel.clj:14:22 - call to method asSegment on jdk.incubator.foreign.MemoryAddress can't be resolved (no such method). Reflection warning, tech/v3/datatype/ffi/nio_buf_mmodel.clj:15:22 - reference to field asByteBuffer can't be resolved. Reflection warning, tech/v3/datatype/ffi/nio_buf_mmodel.clj:19:22 - call to method order can't be resolved (target class is unknown). Reflection warning, tech/v3/datatype/ffi/nio_buf_mmodel.clj:20:19 - call to method order can't be resolved (target class is unknown). Apr 26, 2022 10:12:23 AM libpython-clj2.python.ffi invoke SEVERE: Exception while releasing object java.lang.IllegalArgumentException: No matching method asSegment found taking 2 args for class jdk.internal.foreign.MemoryAddressImpl at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:127) at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:102) at tech.v3.datatype.ffi.nio_buf_mmodel$direct_buffer_constructor.invokeStatic(nio_buf_mmodel.clj:14) at tech.v3.datatype.ffi.nio_buf_mmodel$direct_buffer_constructor.invoke(nio_buf_mmodel.clj:10)...

what am I doing wrong ?

[jjtolton]

Can you paste the full stacktrace? I only see warnings so far. Does it crash?

[Hendekagon]

here:

WARNING: Using incubator modules: jdk.incubator.foreign, jdk.incubator.vector
 Clojure 1.11.1

 (require '[libpython-clj2.require :as pyr :refer [require-python]])
 WARNING: abs already refers to: #'clojure.core/abs in namespace: tech.v3.datatype.functional-api, being replaced by: #'tech.v3.datatype.functional-api/abs
 WARNING: infinite? already refers to: #'clojure.core/infinite? in namespace: tech.v3.datatype.functional-api, being replaced by: #'tech.v3.datatype.functional-api/infinite?
 WARNING: abs already refers to: #'clojure.core/abs in namespace: tech.v3.datatype.functional, being replaced by: #'tech.v3.datatype.functional/abs
 WARNING: infinite? already refers to: #'clojure.core/infinite? in namespace: tech.v3.datatype.functional, being replaced by: #'tech.v3.datatype.functional/infinite?
 Apr 26, 2022 11:43:54 AM libpython-clj2.python.info invoke
 INFO: Detecting startup info
 Apr 26, 2022 11:43:54 AM libpython-clj2.python invoke
 INFO: Startup info {:lib-version "3.8", :java-library-path-addendum "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib", :exec-prefix "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8", :executable "/Library/Developer/CommandLineTools/usr/bin/python3", :libnames ("python3.8m" "python3.8"), :prefix "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8", :base-prefix "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8", :libname "python3.8m", :base-exec-prefix "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8", :python-home "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8", :version [3 8 9], :platform "darwin"}
 Apr 26, 2022 11:43:54 AM libpython-clj2.python invoke
 INFO: Prefixing java library path: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib
 Apr 26, 2022 11:43:55 AM libpython-clj2.python invoke
 INFO: Loading python library: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/libpython3.8.dylib
 Apr 26, 2022 11:43:55 AM tech.v3.resource.gc invoke
 INFO: Reference thread starting
 Apr 26, 2022 11:43:55 AM tech.v3.datatype.nio-buffer invoke
 INFO: Unable to find direct buffer constructor -
 falling back to jdk16 memory model.
 Reflection warning, tech/v3/datatype/ffi/nio_buf_mmodel.clj:14:22 - call to method asSegment on jdk.incubator.foreign.MemoryAddress can't be resolved (no such method).
 Reflection warning, tech/v3/datatype/ffi/nio_buf_mmodel.clj:15:22 - reference to field asByteBuffer can't be resolved.
 Reflection warning, tech/v3/datatype/ffi/nio_buf_mmodel.clj:19:22 - call to method order can't be resolved (target class is unknown).
 Reflection warning, tech/v3/datatype/ffi/nio_buf_mmodel.clj:20:19 - call to method order can't be resolved (target class is unknown).
 Apr 26, 2022 11:43:55 AM libpython-clj2.python.ffi invoke
 SEVERE: Exception while releasing object
 java.lang.IllegalArgumentException: No matching method asSegment found taking 2 args for class jdk.internal.foreign.MemoryAddressImpl
 at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:127)
 at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:102)
 at tech.v3.datatype.ffi.nio_buf_mmodel$direct_buffer_constructor.invokeStatic(nio_buf_mmodel.clj:14)
 at tech.v3.datatype.ffi.nio_buf_mmodel$direct_buffer_constructor.invoke(nio_buf_mmodel.clj:10)
 at clojure.lang.Var.invoke(Var.java:399)
 at tech.v3.datatype.nio_buffer$native_buf__GT_nio_buf.invokeStatic(nio_buffer.clj:113)
 at tech.v3.datatype.nio_buffer$native_buf__GT_nio_buf.invoke(nio_buffer.clj:108)
 at libpython_clj2.python.ffi$pystr__GT_str.invokeStatic(ffi.clj:785)
 at libpython_clj2.python.ffi$pystr__GT_str.invoke(ffi.clj:773)
 at libpython_clj2.python.ffi$pytype_name.invokeStatic(ffi.clj:793)
 at libpython_clj2.python.ffi$pytype_name.invoke(ffi.clj:789)
 at libpython_clj2.python.ffi$pyobject_type_kwd$reify__10514.apply(ffi.clj:809)
 at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
 at libpython_clj2.python.ffi$pyobject_type_kwd.invokeStatic(ffi.clj:805)
 at libpython_clj2.python.ffi$pyobject_type_kwd.invoke(ffi.clj:802)
 at libpython_clj2.python.ffi$wrap_obj_ptr$fn__10517.invoke(ffi.clj:840)
 at libpython_clj2.python.gc$track$fn__9936.invoke(gc.clj:37)
 at tech.resource.GCReference.run(GCReference.java:25)
 at libpython_clj2.python.gc$clear_stack_context.invokeStatic(gc.clj:59)
 at libpython_clj2.python.gc$clear_stack_context.invoke(gc.clj:56)
 at libpython_clj2.python.io_redirect$setup_std_writer$fn__12372.invoke(io_redirect.clj:44)
 at clojure.lang.AFn.applyToHelper(AFn.java:152)
 at clojure.lang.AFn.applyTo(AFn.java:144)
 at clojure.core$apply.invokeStatic(core.clj:667)
 at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1990)
 at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1990)
 at clojure.lang.RestFn.invoke(RestFn.java:425)
 at libpython_clj2.python.io_redirect$setup_std_writer.invokeStatic(io_redirect.clj:44)
 at libpython_clj2.python.io_redirect$setup_std_writer.invoke(io_redirect.clj:40)
 at libpython_clj2.python.io_redirect$redirect_io_BANG_.invokeStatic(io_redirect.clj:54)
 at libpython_clj2.python.io_redirect$redirect_io_BANG_.invoke(io_redirect.clj:52)
 at libpython_clj2.python$initialize_BANG_$fn__12403.invoke(python.clj:107)
 at libpython_clj2.python$initialize_BANG_.invokeStatic(python.clj:100)
 at libpython_clj2.python$initialize_BANG_.doInvoke(python.clj:46)
 at clojure.lang.RestFn.invoke(RestFn.java:397)
 at libpython_clj2.metadata$eval12678.invokeStatic(metadata.clj:17)
 at libpython_clj2.metadata$eval12678.invoke(metadata.clj:17)
 at clojure.lang.Compiler.eval(Compiler.java:7194)
 Execution error (IllegalArgumentException) at tech.v3.datatype.ffi.nio-buf-mmodel/direct-buffer-constructor (nio_buf_mmodel.clj:14).
 No matching method asSegment found taking 2 args for class jdk.internal.foreign.MemoryAddressImpl
 at clojure.lang.Compiler.load(Compiler.java:7653)
 at clojure.lang.RT.loadResourceScript(RT.java:381)
 at clojure.lang.RT.loadResourceScript(RT.java:372)
 at clojure.lang.RT.load(RT.java:459)
 at clojure.lang.RT.load(RT.java:424)
 at clojure.core$load$fn__6908.invoke(core.clj:6161)
 at clojure.core$load.invokeStatic(core.clj:6160)
 at clojure.core$load.doInvoke(core.clj:6144)
 at clojure.lang.RestFn.invoke(RestFn.java:408)
 at clojure.core$load_one.invokeStatic(core.clj:5933)
 at clojure.core$load_one.invoke(core.clj:5928)
 at clojure.core$load_lib$fn__6850.invoke(core.clj:5975)
 at clojure.core$load_lib.invokeStatic(core.clj:5974)
 at clojure.core$load_lib.doInvoke(core.clj:5953)
 at clojure.lang.RestFn.applyTo(RestFn.java:142)
 at clojure.core$apply.invokeStatic(core.clj:669)
 at clojure.core$load_libs.invokeStatic(core.clj:6016)
 at clojure.core$load_libs.doInvoke(core.clj:6000)
 at clojure.lang.RestFn.applyTo(RestFn.java:137)
 at clojure.core$apply.invokeStatic(core.clj:669)
 at clojure.core$require.invokeStatic(core.clj:6038)
 at clojure.core$require.doInvoke(core.clj:6038)
 at clojure.lang.RestFn.invoke(RestFn.java:482)
 at libpython_clj2.require$eval1932$loading__6789__auto____1933.invoke(require.clj:1)
 at libpython_clj2.require$eval1932.invokeStatic(require.clj:1)
 at libpython_clj2.require$eval1932.invoke(require.clj:1)
 at clojure.lang.Compiler.eval(Compiler.java:7194)
 at clojure.lang.Compiler.eval(Compiler.java:7183)
 at clojure.lang.Compiler.load(Compiler.java:7653)
 at clojure.lang.RT.loadResourceScript(RT.java:381)
 at clojure.lang.RT.loadResourceScript(RT.java:372)
 at clojure.lang.RT.load(RT.java:459)
 at clojure.lang.RT.load(RT.java:424)
 at clojure.core$load$fn__6908.invoke(core.clj:6161)
 at clojure.core$load.invokeStatic(core.clj:6160)
 at clojure.core$load.doInvoke(core.clj:6144)
 at clojure.lang.RestFn.invoke(RestFn.java:408)
 at clojure.core$load_one.invokeStatic(core.clj:5933)
 at clojure.core$load_one.invoke(core.clj:5928)
 at clojure.core$load_lib$fn__6850.invoke(core.clj:5975)
 at clojure.core$load_lib.invokeStatic(core.clj:5974)
 at clojure.core$load_lib.doInvoke(core.clj:5953)
 at clojure.lang.RestFn.applyTo(RestFn.java:142)
 at clojure.core$apply.invokeStatic(core.clj:669)
 at clojure.core$load_libs.invokeStatic(core.clj:6016)
 at clojure.core$load_libs.doInvoke(core.clj:6000)
 at clojure.lang.RestFn.applyTo(RestFn.java:137)
 at clojure.core$apply.invokeStatic(core.clj:669)
 at clojure.core$require.invokeStatic(core.clj:6038)
 at clojure.core$require.doInvoke(core.clj:6038)
 at clojure.lang.RestFn.invoke(RestFn.java:408)
 at test.test$eval1926.invokeStatic(test.clj:5)
 at test.test$eval1926.invoke(test.clj:5)
 at clojure.lang.Compiler.eval(Compiler.java:7194)
 at clojure.lang.Compiler.eval(Compiler.java:7149)
 at clojure.core$eval.invokeStatic(core.clj:3215)
 at clojure.core$eval.invoke(core.clj:3211)
 at nrepl.middleware.interruptible_eval$evaluate$fn__968$fn__969.invoke(interruptible_eval.clj:87)
 at clojure.lang.AFn.applyToHelper(AFn.java:152)
 at clojure.lang.AFn.applyTo(AFn.java:144)
 at clojure.core$apply.invokeStatic(core.clj:667)
 at clojure.core$with_bindings_STAR_.invokeStatic(core.clj:1990)
 at clojure.core$with_bindings_STAR_.doInvoke(core.clj:1990)
 at clojure.lang.RestFn.invoke(RestFn.java:425)
 at nrepl.middleware.interruptible_eval$evaluate$fn__968.invoke(interruptible_eval.clj:87)
 at clojure.main$repl$read_eval_print__9206$fn__9209.invoke(main.clj:437)
 at clojure.main$repl$read_eval_print__9206.invoke(main.clj:437)
 at clojure.main$repl$fn__9215.invoke(main.clj:458)
 at clojure.main$repl.invokeStatic(main.clj:458)
 at clojure.main$repl.doInvoke(main.clj:368)
 at clojure.lang.RestFn.invoke(RestFn.java:1523)
 at nrepl.middleware.interruptible_eval$evaluate.invokeStatic(interruptible_eval.clj:84)
 at nrepl.middleware.interruptible_eval$evaluate.invoke(interruptible_eval.clj:56)
 at nrepl.middleware.interruptible_eval$interruptible_eval$fn__999$fn__1003.invoke(interruptible_eval.clj:152)
 at clojure.lang.AFn.run(AFn.java:22)
 at nrepl.middleware.session$session_exec$main_loop__1067$fn__1071.invoke(session.clj:202)
 at nrepl.middleware.session$session_exec$main_loop__1067.invoke(session.clj:201)
 at clojure.lang.AFn.run(AFn.java:22)
 at java.base/java.lang.Thread.run(Thread.java:833)

[cnuernber]

This is a JDK issue. The foreign interface changed again on JDK-18 and I haven't updated dtype-next to handle it. JDK-17 will work iwth your setup I believe.

[Hendekagon]

thank you! that was it
I still get this

INFO: Unable to find direct buffer constructor -
falling back to jdk16 memory model.

but otherwise it now works

[cnuernber]

OK, let's leave this open. I won't fix this for JDK-18 as JDK-19 is coming quite soon and we will need to see what API changes it includes.

[wbridewell]

With JDK-19 released and JDK-20 in early access, has java.lang.foreign stabilized enough to revisit an FFI update?

[cnuernber]

I just landed a PR in dtype-next that purports to do just that :-). I haven't tested it yet.

[cnuernber]

JDK-19 is now supported out of the box with latest libpython-clj :-).