Question - it's possible to use jpype with multiprocessing

[dadokkio]

Hi,
I'm trying to parallelize some jpype code using multiprocessing but I'm not sure if this is possible and if this is the correct way to proceed.

import itertools
import jpype.imports
from jpype import JString
from glob import glob
from multiprocessing import cpu_count, Pool

def write_data(Q):
    (J_Class, condition, parameter) = Q  
    items = J_Class.read(condition, parameter)
    print(len(items.getData()), condition, parameter)

classpath = ":".join(glob("LIB/*.jar"))
jpype.startJVM(jpype.getDefaultJVMPath(), "-Djava.class.path=%s" % classpath, convertStrings=True)
J_API = jpype.JClass("example.class")
J_Class = J_API(JString('filepath'))
param_1 = [int(x) for x in J_Class.getParam1List()]
param_2 = [int(x) for x in J_Class.getParam2List()]

with Pool(cpu_count()) as pool:
    pool.map(write_data, [(J_Class, x, y) for (x, y) in itertools.product(param_1, param_2)])

In this way multiprocessing is not able to pickle the class so I tried to initialize the class directly in the write_data function:

def write_data(Q):
    (condition, parameter) = Q  
    logging.warning(f"This is printed")
    J_API = jpype.JClass("example.class")
    logging.warning(f"This is not printed")
    J_Class = J_API(JString('filepath'))
    items = J_Class.read(condition, parameter)
    print(len(items.getData()), condition, parameter)

with Pool(cpu_count()) as pool:
    pool.map(write_data, [(x, y) for (x, y) in itertools.product(param_1, param_2)])

With this second method I can see all processes spawned but then nothing happens.
Any help?

[Thrameos]

Pickle support was added to the repo a bit ago (#481), but I can’t say how well that will work with the multiprocessing in Python. Some of the tests in the jpypetest directory use primitive types to spawn a job and start a jvm in the remote process, so at least I know that jpype will work in a multiprocess context. Unfortunately, this hasn’t had a lot of focus or testing. So first I would recommend getting the jpype version with pickle support, try your tests again, then if it doesn’t work look over the jpypetest to see how we got multiprocessing working. I believe the leak tester and the startJVM tests both use some sort of multiprocessing.

[dadokkio]

Ok, I'm trying to build newer release from source but now I've new questions :)

• is master the right branch? Is more updated than devel but both seems to be release 0.7.0
• the build is ok, but some test are failing [15 failed, 432 passed, 40 skipped, 2 warnings in 5.93s], Can I ignore them? Do you need the log?

In any case now I've jpype.pickle working but multiprocessing is not able to serialize the class like before.. I need to specify something?

Thanks

[Thrameos]

Master should be the correct branch. And it should not have any fails. Please submit a log with failed tests.

[dadokkio]

I've update the log in this gist: https://gist.github.com/dadokkio/608a6322f2b0fa7054bca83e11ada195
Some machine and setup info:
`[root@xxx1 jpype-master]# ant -v
Apache Ant(TM) version 1.9.2 compiled on January 22 2014

[root@xxx1 jpype-master]# echo $JAVA_HOME
/usr/java/jdk1.8.0_131

[root@xxx1 jpype-master]# python --version
Python 3.7.5

[root@xxx1 jpype-master]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)
`

[Thrameos]

A portion of your test harness is not being built thus it can’t find the java portion of the test. I recommend manually rebuilding the test harness and rerunning the tests. I was not able to replicate your exact symptoms with a clean build. Here is an example of a manual build of the harness ``` $ ant -f test/build.xml Buildfile: C:\Users\nelson85\cygwin\devel\open\test\jpype\test\build.xml compile-main: [javac] C:\Users\nelson85\cygwin\devel\open\test\jpype\test\build.xml:52: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [javac] Compiling 1 source file to C:\Users\nelson85\cygwin\devel\open\test\jpype\test\classes compile-8: [javac] C:\Users\nelson85\cygwin\devel\open\test\jpype\test\build.xml:39: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds compile-9: [javac] C:\Users\nelson85\cygwin\devel\open\test\jpype\test\build.xml:25: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds compile: test: BUILD SUCCESSFUL Total time: 0 seconds ``` Beyond this issue it all looks fine. All of the tests that failed seem likely to pass if the harness is in place as they all are dummy checks to make sure we haven’t broken things on a patch, and all pass on a similarly configured machine.

[dadokkio]

Ok, that's worked!
449 passed, 38 skipped, 2 warnings in 5.70s

But I've the same situation than before:

_pickle.PicklingError: Can't pickle <java class 'example.classes.JFA_API'>: attribute lookup example.classes.JFA_API on jpype._jclass failed

Do I need to specify the pickling class somewhere?

[Thrameos]

The pickler documentation is specified in jpype/pickle.py. When pickling we must create a custom pickler with the java hooks for calling the java serializer. I am not sure how you would pass the custom pickler to the multiclass, but assuming you can then it should all work.

[dadokkio]

Ok, I was able to use the class with getstate and setstate but my problem seems to be related to the class itself because it returns:
jpype._jclass.NotSerializableException: java.io.NotSerializableException

[Thrameos]

All classes that are intended to be serialized need java.lang.Serializable as their base. Unfortunately if this is not true there isn’t a way to push them through that I am aware of.