Unable to start JVM

[keithsun80]

I push the code to the test enviroment , and there are another Exception.... T . T

The project in my enviroment there are no problem.

Exception in below.

Traceback (most recent call last):
  File "/home/prod/softwares/python/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/home/prod/deploys/WiwideOP/libs/rsa/api/rsa_adapter.py", line 63, in wrapper
    result = rsa_api.auth(user_name, pass_code)
  File "/home/prod/deploys/WiwideOP/libs/rsa/api/rsa_api.py", line 26, in auth
    startJVM(getDefaultJVMPath(), "-Djava.class.path=%s" % CLASS_PATH)
  File "build/bdist.linux-x86_64/egg/jpype/_core.py", line 44, in startJVM
    _jpype.startup(jvm, tuple(args), True)
RuntimeError: Unable to start JVM at src/native/common/jp_env.cpp:71

java version "1.6.0_32"
OpenJDK Runtime Environment (IcedTea6 1.13.4) (6b32-1.13.4-4ubuntu0.12.04.2)
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)

We use uwsgi to start project , is that problem?

[keithsun80]

That like as i performed startJVM already, And i perform it again, But i did never do that.

if not isJVMStarted()
    startJVM(getDefaultJVMPath(), "-Djava.class.path=%s" % CLASS_PATH)

In the test enviroment use Python Shell perform startJVM, is no problem,But we use uwsgi start web project and to request it, startJVM throw that Exception.

[keithsun80]

That's so weird, Anyway, If i find solution, i would submit.

[marscher]

I've written a simple testcase e2b128c, which uses two threads calling attachThreadToJVM and it fails, if run in parallel:
https://travis-ci.org/marscher/jpype/jobs/34153184

Not the actual method call of attachThreadToJVM is failing, but the call to a java method afterwards.

@baztian hope thats the right way to handle multiple threads?!

[keithsun80]

Thanks for your replay @marscher .

method_lock = thread.allocate_lock()

def do_sth():
    method_lock.acquire()
    # Other threads will wait until the before thread finished.
    method_lock.release()

I'm not sure, is that could help you? @marscher .

[keithsun80]

Error occurred during initialization of VM
Could not reserve enough space for object heap

[baztian]

Hi @marscher , it seems to be a conflict with system.out accessing the console in parallel somehow. See my attempt for a testcase: baztian@9498a5b
Or my attempt to fix your testcase: baztian@c75adc2

[keithsun80]

i have to change another way.... T . T

[baztian]

@marscher, for some reason, running my test isolated gives me a fatal error.
python test/testsuite.py jpypetest.multithreading.ThreadingTestCase.testParallelThreads

#  SIGSEGV (0xb) at pc=0x00007f0baddb69c8, pid=5151, tid=139687944910592
#
# JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [_jpype.so+0x5a9c8]  JPMethod::findOverload(std::vector<HostRef*, std::allocator<HostRef*> >&, bool)+0x58

Using java version "1.7.0_67"
This also happens if I use your branch and replace the System.out.println(...) with String("foo")

[marscher]

@baztian which operation in this method causes the segfault?

[baztian]

I can reproduce the error only every two or three runs. I can't tell which operation causes the error. But I couldn't reproduce the error if I don't call java code in the thread. In my example it works if I remove the jpype.java.lang.String constructor call.

[marscher]

Well, it sounds like a race condition for the jvm access.
I think all calls to the jvm have to be synchronized somehow. Currently
this has to be handled by the user, which he is not aware of.
We currently do not have any working testcase, where the jvm can be
access in a parallel manner, or am I wrong?

[baztian]

Tried some multithreading stuff with really weird results. Test

    def testSleepWithAttachDetach(self):
        jpype.attachThreadToJVM()
        jpype.java.lang.Thread.sleep(2000)
        jpype.detachThreadFromJVM()

(not in a new python thread) results in

.
----------------------------------------------------------------------
Ran 1 test in 2.104s

OK
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f549581f476, pid=17036, tid=140001294501696
#
# JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [_jpype.so+0x62476]  JPJavaEnv::CallVoidMethod(_jobject*, _jmethodID*)+0x26
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/foo/Sources/jpype/hs_err_pid17036.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#
Running testsuite using JVM /usr/lib/jvm/java-7-oracle/jre//lib/amd64/server/libjvm.so

So the code seems to be executed but at some (unkown) point the process dies.

The following test

    def _java_sleep(self):
        jpype.attachThreadToJVM()
        jpype.java.lang.Thread.sleep(2000)
        jpype.detachThreadFromJVM()

    def testParallelThreads(self):
        t1 = Thread(target=self._java_sleep)
        t2 = Thread(target=self._java_sleep)
        t1.start()
        t2.start()
        t1.join()
        t2.join()

usually results in

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/foo/Sources/jpype/test/jpypetest/multithreading.py", line 28, in _java_sleep
    jpype.java.lang.Thread.sleep(2000)
AttributeError: type object 'java.lang.Thread' has no attribute 'sleep'

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/foo/Sources/jpype/test/jpypetest/multithreading.py", line 28, in _java_sleep
    jpype.java.lang.Thread.sleep(2000)
AttributeError: type object 'java.lang.Thread' has no attribute 'sleep'

.
----------------------------------------------------------------------
Ran 1 test in 0.096s

OK
Running testsuite using JVM /usr/lib/jvm/java-7-oracle/jre//lib/amd64/server/libjvm.so
JVM activity report     :
    classes loaded       : 24

And then stops at this point (I have to kill the process with Ctrl-C). And even weirder: the code sometimes just works.

@marscher, you are right, no (at all time) working test case for jpype multithreading.

[baztian]

Doesn't seem to be a recent regression. JPype1 0.5.4.3 has the same issue.

[baztian]

Was hoping that this commit from @tcalmant using the GIL for something I don't understand could help (see discussion in #18). Unfortunately the test still fails.

[baztian]

BTW: The test only fails if I run it like this:
python test/testsuite.py jpypetest.multithreading
If I run the testsuite completely it succeeds!

[marscher]

sounds very strange. It succeeded in how many runs? We already noticed
random behavior...

[baztian]

I did not manage to make the test fail if I run the whole suite.
Running only jpypetest.multithreading succeeds about every fifth run.
(I'm talking about the testcase in my branch:

    def _java_printer(self, name, q):
        jpype.attachThreadToJVM()
        q.put(jpype.java.lang.String("foobar %s" % name))
        jpype.detachThreadFromJVM()

    def testParallelThreads(self):
        q = Queue()
        t1 = Thread(target=self._java_printer, args=("thread 1", q))
        t2 = Thread(target=self._java_printer, args=("thread 2", q))
        t1.start()
        t2.start()
        t1.join()
        t2.join()
        results = q.get(), q.get()
        self.assertItemsEqual(results, ["foobar thread 1",
                                        "foobar thread 2"])

)

[marscher]

well that is at least a starting point to investigate this. But very
strange either. The only way the invocation differs in execution logic
is, that we might actually import the testcases via the selective method.

[baztian]

I guess it has something to do with the state of the jpype module. If I change the name multithreading.py to abmultithreading.py it get's executed first. And now it fails with the same rate as with the selective method.

[marscher]

This intuition should be perfectly right. But how the hell we can pin
this down? Have to think about it.

[rapgro]

The discussion here leads me to a more general question. Is JPype designed and able to execute multiple threads? Are those threads executed in the same JVM instance or in several processes individually? I've seen strange issues with Java when there are two JVM instances running in parallel.

[baztian]

JPype is able to execute Java code from multiple Python threads or multiple Java threads. In the first case make sure you call attachThreadToJVM and detachThreadToJVM in the corresponding Python threads. JPype only allows one JVM instance during one Python interpreter session.

[StevenLOL]

same error, to reproduce:
'''
startJVM(getDefaultJVMPath(), "-ea")
shutdownJVM()
startJVM(getDefaultJVMPath(), "-ea")
shutdownJVM()
'''
Java 7 Oracle, UBUNTU 15.04 64bits , python 2.7

[marscher]

@StevenLOL according to the fine documentation, "restarts" are not supported by the JVM.

[StevenLOL]

@marscher , thanks!

[Harshanah]

I got around this problem by creating a shell script . When program finished JVM can be shutdown . then parameters are passed from the shell script
EX
#!/bin/bash
for (( i=5; i<=21; i++ ))
do
k=0;
python use.py $i# i is passed as a system argument to python
echo "Finished $i"
done