rJava fails to deal with JVMTI

[mohanr]

I use rJava and call Java code that uses JVMTI code like this

   VM vm = VM.getVM();
    Universe universe = vm.getUniverse();
     CollectedHeap heap = universe.heap();

rJava does not call this code directly but calls a method that in turn internally uses JVMTI.

This code attaches to another JVM and to get its internals. This is legal Java and it works when tested separately. But when I use rJava to call this code, the call is successful but it throws an error. I believe there is Java native Interface involved in the JVM and it seems to affect R. rJava is 0.9

version
_
platform x86_64-apple-darwin10.8.0
arch x86_64
os darwin10.8.0
system x86_64, darwin10.8.0
status
major 3
minor 1.0
year 2014
month 04
day 10
svn rev 65387
language R
version.string R version 3.1.0 (2014-04-10)
nickname Spring Dance

Error: C stack usage 140730070087404 is too close to the limit
Error: C stack usage 140730070156700 is too close to the limit
Warning: stack imbalance in '.Call', 59 then -1
Warning: stack imbalance in '{', 56 then -4

*** caught bus error ***
address 0x100583fd8, cause 'non-existent physical address'

*** caught bus error ***
address 0x100583fd8, cause 'non-existent physical address'

Traceback:
1: run(timeoutMs)
2: service(timeout)

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace
Selection: run(10)

*** caught segfault ***
address 0x20057ea40, cause 'memory not mapped'

Traceback:
1: run(timeoutMs)
2: service(timeout)

Possible actions:
1: abort (with core dump, if enabled)
2: normal R exit
3: exit R without saving workspace
4: exit R saving workspace

[s-u]

This typically means that the invoked Java code has manipulated the C stack which is illegal. One possibility is that this happens because JVMTI itself uses JNI which in turn messes up the C stack. One common way that leads to stack relocations is the use of threads in an unsafe manner - i.e. the native code returns on another thread with a different C stack. In that case you should modify your code such that it is guaranteed to return to R on the same thread (if necessary defer all code into a separate thread that will never interact with R).

Another (unsafe) possibility is to disable stack monitoring in R, by setting R_CStackLimit to -1 but that should be only a desperate measure, because any recursion-related errors (including regular R evaluation) will lead to segfaults.

[mohanr]

I have updated this comment and removed references to JVMTI. This is the serviceability Java API.

I did come across R_CStackLimit but it may not be the solution.

I couldn't understand the link between the C stack and threads. As far as my Java code is concerned the serviceability API call is executed in a new thread.

                Future<Void> captureYoungCapacity =
                    es.submit(  new Callable<Void>(){
                    @Override
                    public Void call()
                    {
                        tool.capture(pid);//Serviceability
                        tool.stopCapture();//Serviceability
                        return null;
                    }
                    } );
                    es.shutdown();

Sorry. I have minimal experience with JNI and C stacks. But if you can explain the problem in a way that the Java serviceability forum can understand I can post it there.

                    metaSpaceObserver <- .jnew("com/rxjava/jmx/MetaSpaceObserver")
                   .jcall(metaSpaceObserver,"V","consume")

Do you mean that even this code(constructor call and method call) should be spawned as a separate thread ?

[s-u]

Each thread has typically its own C stack. Therefore when you enter Java on one thread and then return back on another thread, it will have a different stack and thus all hell breaks loose. Therefore you must make sure that you return on the same thread that you entered. In some circumstances you may have a special design where you lock the first thread and thus it may be possible to switch threads, but then you must disable stack limit since you'll return on another stack. It is, however, safer to add proper synchronization and make sure you stay on the same thread.

Well, I can only explain what the error says (see above), but how it happens is entirely up to you since you didn't provide any reproducible example (the code above fails with ClassNotFoundException in regular Java). The rule is simple: you have to guarantee that you don't switch threads or stacks when you return back to R.

[mohanr]

Delayed response :
I think this is reproducible. sda-jdi.jar part the JDK in the 'lib' folder, is required to build and execute.
This is the SA API and not JVMTI as originally mentioned.

import sun.jvm.hotspot.gc_implementation.parallelScavenge.PSYoungGen;
import sun.jvm.hotspot.gc_implementation.parallelScavenge.ParallelScavengeHeap;
import sun.jvm.hotspot.gc_implementation.shared.MutableSpace;
import sun.jvm.hotspot.gc_interface.CollectedHeap;
import sun.jvm.hotspot.memory.Universe;
import sun.jvm.hotspot.runtime.VM;
import sun.jvm.hotspot.tools.Tool;

public class YoungGenObserverData extends Tool{
private long youngCapacity;

@Override
public void run() {
   VM vm = VM.getVM();
   Universe universe = vm.getUniverse();
    CollectedHeap heap = universe.heap();
    if (heap instanceof ParallelScavengeHeap) {
        ParallelScavengeHeap psHeap = (ParallelScavengeHeap) heap;
        PSYoungGen y = psHeap.youngGen();
        MutableSpace youngObjSpace = y.edenSpace();
        youngCapacity = youngObjSpace.capacity();
    }
}

}

[s-u]

The above is incomplete since something has to load and run the tool and that's likely where the problem is (just calling run() from R yield VM.initialize() was not yet called). This API has been very badly documented and AFAICS has removed from latest JDKs so I'm closing this as probably nothing to do with rJava.