Make JNA Binding available to Java clients

[subes]

As discussed here: cnuernber/libjulia-clj#3
Please generate some Java classes for libpython-clj so I can integrate it here: https://github.com/invesdwin/invesdwin-context-python/tree/master/invesdwin-context-python-parent/invesdwin-context-python-runtime-libpythonclj

Thanks a lot!

[cnuernber]

This is great :-). Is this going to be used in the same classloader as the libjulia-clj java bindings?

The reason I ask is because if so, then I need to release a dtype-next aot version and make the libjulia and this python version dependent on that so that we don't duplicate class files all over the place - the libjulia aot version I made had the required dtype-next class files generated inline.

For the first test I can just regenerate the dtype-next classes but after we confirm this is working I will need to release aot versions of dependent libraries if this is going to be used in the same classloader as the libjulia-clj java bindings.

[cnuernber]

Initial jar - https://clojars.org/clj-python/libpython-clj/versions/2.004-aot
API docs - https://clj-python.github.io/libpython-clj/libpython-clj2.java-api.html

The initialize function calls the python executable from command line and gets it to output the setup information. Python has a bit more involved setup than just a shared library path as it also needs to know some level of module root path. Let's see if this works and we can tweak.

Small unit test showing functionality - https://github.com/clj-python/libpython-clj/blob/master/test/libpython_clj2/java_api_test.clj

[subes]

Thanks a lot, I will test it as soon as possible.

• Yes both Julia and Python should be able to be used in the same JVM process.
• Is there an option so that I define a different path to the python executable to be used? I would like to make this configurable so it does not need to be in the PATH. Though the default value would be "python3" or something like that from the PATH. => found the settings for that already: https://clj-python.github.io/libpython-clj/libpython-clj2.python.html#var-initialize.21

[subes]

I made some progress here: https://github.com/invesdwin/invesdwin-context-python/blob/master/invesdwin-context-python-parent/invesdwin-context-python-runtime-libpythonclj/src/main/java/de/invesdwin/context/python/runtime/libpythonclj/internal/PythonEngine.java

Though I can't figure out how to get/set globals. I get this exception:

2021-12-28 14:59:13.994 [ |7-7:InputsAndResult] ERROR de.invesdwin.ERROR.process                                   - processing #00000007
de.invesdwin.context.log.error.LoggedRuntimeException: #00000007 clojure.lang.ArityException: Wrong number of args (3) passed to: libpython-clj2.python/get-item
        ... 12 omitted, see following cause or error.log
Caused by - clojure.lang.ArityException: Wrong number of args (3) passed to: libpython-clj2.python/get-item
        at clojure.lang.AFn.throwArity(AFn.java:429)
        at clojure.lang.AFn.invoke(AFn.java:40)
        at libpython_clj2.java_api$_setItem.invokeStatic(java_api.clj:106)
        at libpython_clj2.java_api$_setItem.invoke(java_api.clj:104)
        at libpython_clj2.java_api.setItem(Unknown Source)
      * at de.invesdwin.context.python.runtime.libpythonclj.internal.PythonEngine.set(PythonEngine.java:48) *
      * at de.invesdwin.context.python.runtime.libpythonclj.LibpythoncljScriptTaskInputsPython.putString(LibpythoncljScriptTaskInputsPython.java:57) *
      * at de.invesdwin.context.python.runtime.contract.hello.HelloWorldScript$1.populateInputs(HelloWorldScript.java:29) *
      * at de.invesdwin.context.python.runtime.libpythonclj.LibpythoncljScriptTaskRunnerPython.run(LibpythoncljScriptTaskRunnerPython.java:42) *
      * at de.invesdwin.context.python.runtime.contract.AScriptTaskPython.run(AScriptTaskPython.java:12) *
      * at de.invesdwin.context.python.runtime.contract.hello.HelloWorldScript.testHelloWorld(HelloWorldScript.java:45) *
      * at de.invesdwin.context.python.runtime.contract.InputsAndResultsTests.test(InputsAndResultsTests.java:23) *
      * at de.invesdwin.context.python.runtime.contract.InputsAndResultsTests$1.run(InputsAndResultsTests.java:47) *
      * at de.invesdwin.util.concurrent.internal.WrappedRunnable.run(WrappedRunnable.java:47) *
        ... 8 more, see error.log

getAttr/setAttr also does not work:

2021-12-28 15:03:03.024 [ |7-2:InputsAndResult] ERROR de.invesdwin.ERROR.process                                   - processing #00000002
de.invesdwin.context.log.error.LoggedRuntimeException: #00000002 java.lang.Exception: AttributeError: 'dict' object has no attribute 'hello'

        ... 12 omitted, see following cause or error.log
Caused by - java.lang.Exception: AttributeError: 'dict' object has no attribute 'hello'

        at libpython_clj2.python.ffi$check_error_throw.invokeStatic(ffi.clj:687)
        at libpython_clj2.python.ffi$check_error_throw.invoke(ffi.clj:685)
        at libpython_clj2.python.base$fn__10297.invokeStatic(base.clj:144)
        at libpython_clj2.python.base$fn__10297.invoke(base.clj:114)
        at libpython_clj2.python.protocols$fn__10100$G__10095__10109.invoke(protocols.clj:57)
        at libpython_clj2.python.bridge_as_jvm$generic_python_as_map$reify__10853$fn__10856.invoke(bridge_as_jvm.clj:281)
        at libpython_clj2.python.bridge_as_jvm$generic_python_as_map$reify__10853.set_attr_BANG_(bridge_as_jvm.clj:281)
        at libpython_clj2.python$set_attr_BANG_$fn__11372.invoke(python.clj:199)
        at libpython_clj2.python$set_attr_BANG_.invokeStatic(python.clj:199)
        at libpython_clj2.python$set_attr_BANG_.invoke(python.clj:196)
        at libpython_clj2.java_api$_setAttr.invokeStatic(java_api.clj:85)
        at libpython_clj2.java_api$_setAttr.invoke(java_api.clj:82)
        at libpython_clj2.java_api.setAttr(Unknown Source)
      * at de.invesdwin.context.python.runtime.libpythonclj.internal.PythonEngine.set(PythonEngine.java:48) *
      * at de.invesdwin.context.python.runtime.libpythonclj.LibpythoncljScriptTaskInputsPython.putString(LibpythoncljScriptTaskInputsPython.java:57) *
      * at de.invesdwin.context.python.runtime.contract.hello.HelloWorldScript$1.populateInputs(HelloWorldScript.java:29) *
      * at de.invesdwin.context.python.runtime.libpythonclj.LibpythoncljScriptTaskRunnerPython.run(LibpythoncljScriptTaskRunnerPython.java:42) *
      * at de.invesdwin.context.python.runtime.contract.AScriptTaskPython.run(AScriptTaskPython.java:12) *
      * at de.invesdwin.context.python.runtime.contract.hello.HelloWorldScript.testHelloWorld(HelloWorldScript.java:45) *
      * at de.invesdwin.context.python.runtime.contract.InputsAndResultsTests.test(InputsAndResultsTests.java:23) *
      * at de.invesdwin.context.python.runtime.contract.InputsAndResultsTests$1.run(InputsAndResultsTests.java:47) *
      * at de.invesdwin.util.concurrent.internal.WrappedRunnable.run(WrappedRunnable.java:47) *
        ... 8 more, see error.log

I also tried just using "globals" as a string instead of using the returned value from the persistent map from runString, but that fails also:

2021-12-28 15:04:17.408 [ |7-10:InputsAndResul] ERROR de.invesdwin.ERROR.process                                   - processing #00000010
de.invesdwin.context.log.error.LoggedRuntimeException: #00000010 clojure.lang.ArityException: Wrong number of args (3) passed to: libpython-clj2.python/get-item
        ... 12 omitted, see following cause or error.log
Caused by - clojure.lang.ArityException: Wrong number of args (3) passed to: libpython-clj2.python/get-item
        at clojure.lang.AFn.throwArity(AFn.java:429)
        at clojure.lang.AFn.invoke(AFn.java:40)
        at libpython_clj2.java_api$_setItem.invokeStatic(java_api.clj:106)
        at libpython_clj2.java_api$_setItem.invoke(java_api.clj:104)
        at libpython_clj2.java_api.setItem(Unknown Source)
      * at de.invesdwin.context.python.runtime.libpythonclj.internal.PythonEngine.set(PythonEngine.java:48) *
      * at de.invesdwin.context.python.runtime.libpythonclj.LibpythoncljScriptTaskInputsPython.putString(LibpythoncljScriptTaskInputsPython.java:57) *
      * at de.invesdwin.context.python.runtime.contract.hello.HelloWorldScript$1.populateInputs(HelloWorldScript.java:29) *
      * at de.invesdwin.context.python.runtime.libpythonclj.LibpythoncljScriptTaskRunnerPython.run(LibpythoncljScriptTaskRunnerPython.java:42) *
      * at de.invesdwin.context.python.runtime.contract.AScriptTaskPython.run(AScriptTaskPython.java:12) *
      * at de.invesdwin.context.python.runtime.contract.hello.HelloWorldScript.testHelloWorld(HelloWorldScript.java:45) *
      * at de.invesdwin.context.python.runtime.contract.InputsAndResultsTests.test(InputsAndResultsTests.java:23) *
      * at de.invesdwin.context.python.runtime.contract.InputsAndResultsTests$1.run(InputsAndResultsTests.java:47) *
      * at de.invesdwin.util.concurrent.internal.WrappedRunnable.run(WrappedRunnable.java:47) *
        ... 8 more, see error.log

[subes]

Ah, this is a lot simpler than expected, one just has to put/get from/into the PersistentMap.

[cnuernber]

True, but there isn't a way to get the globals map in a stand-alone fashion which I can see would be useful. And you did find an error in the api :-).

I tried to expose the python objects as their java equivalents so a python dict will be returned as an implementation of java.util.Map and tuples/lists implement java.util.List and java.util.RandomAccess, etc. There is a somewhat complex pathway in there in Clojure for if you want to copy a python value completely into the JVM such as copying a JSON object or if you want to bridge/proxy it like what people want to do with modules.

[cnuernber]

Also I want to expose a withGIL function so you can capture the GIL once and do a set of things. This is similar to inContext with the exception that it doesn't attempt to release all objects allocated within the scope.

[subes]

This is how I get the globals right now:

    @SuppressWarnings("unchecked")
    private Map<Object, Object> getGlobals() {
        final Map<?, ?> mainModule = libpython_clj2.java_api.runString("");
        return (Map<Object, Object>) mainModule.get("globals");
    }

[cnuernber]

Yep, that will work.

[subes]

And yes, withGIL would be great. Though a lock/unlock function would be better so I can do:

gilLock.lock();
try{
...
} finally{
    gilLock.unlock();
}

[cnuernber]

OK, makes sense. Then you can make your own withGIL if you need it...

[subes]

I would just wrap it in an implementation of ILock (which my client code already uses).

[cnuernber]

New API is up - it has two new functions for GIL management (as well as a fixed setItem call) - https://clojars.org/clj-python/libpython-clj/versions/2.004-aot-1.

Note that the python ensureGIL call returns an integer that must be passed into releaseGIL so ensureGIL is 'reentrant' but you would have to keep track of that integer in your ILock impl.

New api fns are int lockGIL() and void unlockGIL(int).

[subes]

Why doesn't this:

final Map<String, Object> initParams = new HashMap<>();
initParams.put("python-executable", "someWrongCommand");
libpython_clj2.java_api.initialize(initParams);

throw an exception? I have no PYTHON_HOME env variable set.
Also I guess python2 and pypy are not supported?