Exception chaining/cause not mirrored into Python

[pelson]

Both Python and Java have exception chaining (Throwable.getCause() in Java, exception.__cause__ in Python).

This information doesn't make it across from Java to Python though. Take the following snippet:

import jpype as jp
jp.startJVM()


java = jp.JPackage('java')
e1 = java.lang.RuntimeException("Some parent exception")
e2 = java.lang.RuntimeException("Some message", e1)

When we inspect the object:

>>> print(e2.__cause__, e2.__context__)
None None

>>> raise e2
Traceback (most recent call last):
  File "/home/pelson/github/jpype/jpype/support/exception_cause.py", line 19, in <module>
    raise e2
java.lang.RuntimeException: java.lang.RuntimeException: Some message

By adding the exception to __cause__ (via e2.__cause__ = e1) we get an improved exception:

java.lang.RuntimeException: java.lang.RuntimeException: Some parent exception

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/pelson/github/jpype/jpype/support/exception_cause.py", line 19, in <module>
    raise e2
java.lang.RuntimeException: java.lang.RuntimeException: Some message

In investigating this, I also wondered if I could pass a Python exception as a cause for a Throwable, but unfortunately that wasn't possible:

>>> e1 = ValueError('foo')
>>> e2 = java.lang.RuntimeException("Some message", e1)

TypeError: No matching overloads found for constructor java.lang.RuntimeException(str,ValueError), options are:
	public java.lang.RuntimeException()
	public java.lang.RuntimeException(java.lang.String)
	public java.lang.RuntimeException(java.lang.String,java.lang.Throwable)
	public java.lang.RuntimeException(java.lang.Throwable)

I'm not sure on the practicalities, but it would be also interesting if we could make BaseException be understood as a Throwable.

[Thrameos]

The Python exception system is mostly hardcoded which makes extensions of it very complicated. You can see the level of hacking that was required to get the stack frames to connect to each other. I have some outstanding tickets with the Python developers regarding providing methods to connect the two together but as I can't contribute to Python (seeing as my employer refuses to sign the contributor agreement) I remain stuck.

As for __cause__ that may be possible, if that field is not part of the hard coding. My main concern is that I use the __cause__ field when chaining on from Python to Java. But I would need to look more closely to see if the Java wrapper can override that field. The unfortunate side effect of the way that Python has implemented this, is that even if I were to make __cause__ to be the correct value when probed, the Python exception printer ignores user side code and simply goes to the hard coded slot. Thus to make it actually work properly with the printers I would have to convert the __cause__ object to Python (and its __cause__ etc, proactively whenever an Exception wrapper was created. This means a speed hit for a bunch of information which may get thrown away without ever being looked at. Of course if you are Java oriented and "Exceptions should be exceptional" rather than Pythonic "Exceptions are just another return path", then the speed hit should be rare.

As for making a ValueError appear as a BaseException. This is possible with the reverse bridge. The only issue it that we still having not recruited enough people to serve as beta testers/developers to complete the epypj code. I know that writing 1000 test routines and solving the pile of edge cases are not fun tasks, but it would be required to release that code.

[Thrameos]

I looked into this some further. It is possible for me to wire up the __cause__, however this creates a conflict in the stack traces produced by Java and the ones produced within Python. The issue is that CPython does not currently let use add the Java stackframes into the exception stack, so we are currently creating a fake exception with the Java stackframes and placing it as the cause. For this to be a meaningful PR, we would need to find someway to insert a marker into the Python traceback system such that when an exception is thrown, we would know where in the Python stack trace listing the Java call occurred, and then replace the marker with the Java stackframes that we are getting from the Java exception.

I see that that whole system CFrame system is being reworked from 3.11 so it is very unclear if there is any way this can be accomplished in a cross version fashion. At least as far as I can tell, there is no public API for creating tracebacks (so we are hacking that in place), and there is no way in the public API to push a marker into the Python traceback and pop it after the Java call returns such that if an exception took place in Java that we can splice out exception information in. However, perhaps @markshannon be so kind as pointing me towards the API if it exists.

[ryanmkurtz]

I am currently trying to get PyGhidra to output the full Java stacktrace when a I detect a JException has occurred from the Python side of things. Appending JException.stacktrace() to the output mostly does the trick, but if the JException has a cause, that isn't included in the stracktrace() string. Is this ticket talking about the same issue?

[ryanmkurtz]

I'm wondering if I should make a new ticket to to include the "cause" in the stacktrace() string?

[Thrameos]

I believe that stacktrace is in the Python layer which should be patchable easily.

    # Included for compatibility with JPype 0.6.3
    def stacktrace(self):
        """ Get a string listing the stack frame.

        Returns:
          A string with the classic Java ``printStackTrace`` result.
        """
        StringWriter = _jpype.JClass("java.io.StringWriter")
        PrintWriter = _jpype.JClass("java.io.PrintWriter")
        sw = StringWriter()
        pw = PrintWriter(sw)
        self.printStackTrace(pw)
        pw.flush()
        r = sw.toString()
        sw.close()
        return r

Are you sure that the cause is not attached? According the Java documentation it should be included.

Test.java

public class Test {
    public void go() {
        try {
            // Level 1: The root cause
            throw new IllegalArgumentException("Root cause error");
        } catch (IllegalArgumentException rootCause) {
            // Level 2: Wrapping the root cause in a new exception
            throw new RuntimeException("Wrapped by RuntimeException", rootCause);
        }
    }
}

test.py:

import jpype
jpype.startJVM(classpath=".")
try:
    jpype.JClass("Test")().go()
except Exception as ex:
    print(ex.stacktrace())

Output:

java.lang.RuntimeException: Wrapped by RuntimeException
        at Test.go(Test.java:8)
Caused by: java.lang.IllegalArgumentException: Root cause error
        at Test.go(Test.java:5)

So unfortunately I can't fix what I can't reproduce. I recommend checking to see if something in the type chain has perhaps overridden printStackTrace.

[ryanmkurtz]

I am seeing the cause now, thanks. But the JPype output (left) looks a bit different than its pure-Java equivalent (right). The true cause seems to be replaced by PyExceptionProxy, and the message on the first line seems wrong.

[ryanmkurtz]

Here is some context for those 2 outputs

Left red output (JPype used):

from jpype import JImplements, JOverride
from ghidra.pcode.exec.PcodeExecutorStatePiece import Reason
from ghidra.pcode.exec import (
    BytesPcodeArithmetic, PcodeExecutor, PcodeExecutorState, PcodeUseropLibrary, SleighProgramCompiler)
print("Test")

@JImplements(PcodeExecutorState)
class MyState():
    @JOverride
    def getLanguage(self, *args):
        raise Exception("Blargh")

    @JOverride
    def getArithmetic(self, *args):
        raise Exception("Blargh")

    @JOverride
    def getVar(self, *args):
        raise Exception("Blargh")

    @JOverride
    def getVarInternal(self, *args):
        raise Exception("Blargh")

    @JOverride
    def getConcreteBuffer(self, *args):
        raise Exception("Blargh")

    @JOverride
    def setVar(self, *args):
        raise Exception("Blargh")

    @JOverride
    def setVarInternal(self, *args):
        raise Exception("Blargh")

    @JOverride
    def getRegisterValues(self, *args):
        raise Exception("Blargh")

    @JOverride
    def fork(self, *args):
        raise Exception("Blargh")

    @JOverride
    def clear(self, *args):
        raise Exception("Blargh")

    @JOverride
    def streamPieces(self, *args):
        raise Exception("Blargh")


language = currentProgram.language

st = MyState()
arithmetic = BytesPcodeArithmetic.forLanguage(language)
pcex = PcodeExecutor(language, arithmetic, st, Reason.EXECUTE_READ)
prog = SleighProgramCompiler.compileProgram(
    language, "Test", "RIP = 0xdeadbeef;", PcodeUseropLibrary.NIL)

pcex.execute(prog, PcodeUseropLibrary.NIL)

Right output (pure java)

import java.util.Map;
import java.util.stream.Stream;

import ghidra.app.plugin.processors.sleigh.SleighLanguage;
import ghidra.app.script.GhidraScript;
import ghidra.pcode.exec.*;
import ghidra.pcode.exec.PcodeArithmetic.Purpose;
import ghidra.pcode.exec.PcodeExecutorStatePiece.Reason;
import ghidra.program.model.address.Address;
import ghidra.program.model.address.AddressSpace;
import ghidra.program.model.lang.Language;
import ghidra.program.model.lang.Register;
import ghidra.program.model.mem.MemBuffer;

public class TestGP6664Script extends GhidraScript {

        static class MyState implements PcodeExecutorState<byte[]> {

                @Override
                public Language getLanguage() {
                        throw new RuntimeException("Blargh");
                }

                @Override
                public PcodeArithmetic<byte[]> getArithmetic() {
                        throw new RuntimeException("Blargh");
                }

                @Override
                public Stream<PcodeExecutorStatePiece<?, ?>> streamPieces() {
                        throw new RuntimeException("Blargh");
                }

                @Override
                public void setVar(AddressSpace space, byte[] offset, int size, boolean quantize,
                                byte[] val) {
                        throw new RuntimeException("Blargh");
                }

                @Override
                public void setVarInternal(AddressSpace space, byte[] offset, int size, byte[] val) {
                        throw new RuntimeException("Blargh");
                }

                @Override
                public byte[] getVar(AddressSpace space, byte[] offset, int size, boolean quantize,
                                Reason reason) {
                        throw new RuntimeException("Blargh");
                }

                @Override
                public byte[] getVarInternal(AddressSpace space, byte[] offset, int size, Reason reason) {
                        throw new RuntimeException("Blargh");
                }

                @Override
                public Map<Register, byte[]> getRegisterValues() {
                        throw new RuntimeException("Blargh");
                }

                @Override
                public MemBuffer getConcreteBuffer(Address address, Purpose purpose) {
                        throw new RuntimeException("Blargh");
                }

                @Override
                public void clear() {
                        throw new RuntimeException("Blargh");
                }

                @Override
                public PcodeExecutorState<byte[]> fork(PcodeStateCallbacks cb) {
                        throw new RuntimeException("Blargh");
                }
        }

        @Override
        protected void run() throws Exception {
                var language = (SleighLanguage) currentProgram.getLanguage();

                var st = new MyState();
                var arithmetic = BytesPcodeArithmetic.forLanguage(language);
                var pcex = new PcodeExecutor<>(language, arithmetic, st, Reason.EXECUTE_READ);
                var prog = SleighProgramCompiler.compileProgram(language, "Test", "RIP = 0xdeadbeef;",
                        PcodeUseropLibrary.nil());

                pcex.execute(prog, PcodeUseropLibrary.nil());
        }
}

[ryanmkurtz]

Seems like the root of the problem is calling raise Exception("Blargh") instead of raise java.lang.RuntimeException("Blargh").

If I raise a pure Java exception, i get the desired strack traces and messages. But if the user raises a Python exception, some info seems be lost.

Is this behavior difference by design?

[Thrameos]

One is a Python exception which captures only the python state, the other is a java exception that will trigger both sides of the bridge to dump their stack traces. Unfortunately, I think that is inherent unless we get a throw on the Java side via a proxy. Right now the proxy just passes the python exception straight through without the java state.

Now that I have a reproduced I can look to see whether there is a clean way to add the cause chain after it comes back down. I suspect this is something that may be possible with reverse bridge, but may be limited by the CPython api.

The place you are looking for is in native/common where the wrapped exception came back down or was sent up. There is a brief moment where we may have both copies of the exception info. Jpexception.cpp toJava exception would be the place I would start as that is the point that triggers java to wrap python triggered. That is the point where you can attach a cause. But is is a nasty place as you must consider if we have called through more than once to avoid recursively attaching.

If I get a chance I can analyze the specific path.
The easiest short term fix may be to use JOverrides to insert a capture and convert point. I would say it may be possible through requires careful planning if it was a feature. Also this may want to wait until after reverse bridge as there has much cleaner paths as you can call back from java to make the attachment.

[ryanmkurtz]

Ok, thank you for taking a look!

[pelson]

I'm not convinced this should be a discussion - it is a limitation in the JPype implementation that we can overcome in the future. Perhaps @Thrameos you have a desire to discuss and agree on a shape of a solution, before we formalise it as a feature request?