'this' is not available - when enabling mock-maker-inline

[leonroy]

When we enable mock-maker-inline in our project and debug a class with @Spy annotated on it IntelliJ reports that 'this' is not available:

When we unset mock-maker-inline then things work as expected and we can inspect the class variables:

It appears to be the same as this report: https://stackoverflow.com/questions/62661996/unable-to-debug-junit-test-anywhere

Is this expected Mockito behaviour?

[TimvdLippe]

Thanks for filing the issue. Could you please provide us a minimal reproduction case that we can clone/download to debug the issue? Thanks in advance!

[AlarkaSanyal]

Facing a similar issue. It can be reproduced by following these steps: https://www.baeldung.com/mockito-final
and then debugging through the code. For any variable, in the Variables window, we can see the following

[jwb441]

I am able to recreate this in Eclipse with the following example
Eclipse: 2020-09 (4.17.0)
Java: jdk1.8.0_201
JUnit: 5.4.2
mockito: 3.6.0 (core buildtime inline testruntime)

When I try to step into the call at
assertEquals(1, myClass.doAThing());

I get: com.sun.jdi.InternalException: Got error code in reply:35 occurred retrieving 'this' from stack frame.

MockitoInlineTest.txt
MyClass.txt

[jwb441]

Let me know if there is any additional data I can add to help get to the bottom of this.

Thanks!
John

[afillatre]

Same issue with Mockito-inline 3.8.0 and java 15. This makes tests very difficult to debug, because it makes impossible to debug the code step by step

[MojoQ]

Our team is having this issue as well with 3.9.0. It's a major show-stopper. Would be great if it'd be fixed <3

[TimvdLippe]

Rafael, does this mean we need to generate some more bytecode to make sure that this references are working as intended with the inline mockmaker?

[raphw]

I am afraid we are the wrong place to fix this. Byte Buddy needs to shift the local variable array to add Mockito's logic. For reasons (tm), this requires to move the this variable further to the left. It is however still there:

  LocalVariableTable:
    Start  Length  Slot  Name   Signature
       76       4     2  this   Lfoo/Sample;

IntelliJ could certainly pick this up and make the variable available in the debugger. Normally this is put at slot 0 but the JVM does not require this at all. I think IntelliJ falsely assumes this. If I rename the variable to this2, it is even displayed but that would be inconsistent for other parsing processors. This needs to be fixed in IntelliJ.

[TimvdLippe]

Thank you for the explanation! Does anybody want to help us out filing this bug for IntelliJ?

[jwb441]

Would something similar need to be done to get this fixed in eclipse?

thank you for looking at it!

[raphw]

Possibly, I have not used Eclipse in a while. This might even relate to the debugger API.

[alblue]

As per twitter conversation:

The JVM spec says that for instance methods, the "this" reference is sorted in slot 0 on a stack frame:

https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-2.html#jvms-2.6.1

"On instance method invocation, local variable 0 is always used to pass a reference to the object on which the instance method is being invoked (this in the Java programming language)."

This is hard coded into the Java debug protocol which is why IDEs like IntelliJ and Eclipse report specific issues with "this" but not other variables; there's a specific "thisObject" call in the wire protocol (as opposed to a local variable which happens to have the name "this"

https://github.com/openjdk/jdk/blob/master/src/jdk.jdwp.agent/share/native/libjdwp/StackFrameImpl.c#L378

https://docs.oracle.com/en/java/javase/11/docs/api/jdk.jdi/com/sun/jdi/StackFrame.html#thisObject

[raphw]

Why do IDEs not process local variables named 'this' though? It's the very same value, would not make a difference for the user and the verifier certainly considers the byte code legal.

It is certainly true that variables are passed on their respective index but they are commonly reassigned within the method what is what happens here.

[alblue]

Because in Java, at least, you can't create a variable called "this" and in fact it's a synthetic construct generated by the java compiler as a shorthand for local[0].

Secondly the local variable table is optional which can be skipped when generating code using -g:none. In this circumstance it is necessary for the debugger to be able to resolve what the object instance is instead of looking up by name.

Thirdly it's what is defined in the debug api which is what IDEs use and which is baked into various assumptions and places within the JVM, including various JIT related operations. This may not be an issue in this case (particularly if it's not called frequently) but for performance reasons the calls to other instance methods will passthrough local[0] implicitly, regardless of what the local variable table has in it.

TL;DR "this" isn't a variable in the normal sense and changing the LVT isn't sufficient to make IDEa believe that it is.

[raphw]

'this' only exists as a Java keyword and you are feee to call it whatever in the language, you like, also in the LVT. The zero slot is in no way privileged. The verifier already tracks if the slot is reassigned to avoid super calls on other identities then the currently dispatched instance. At the same time, it allows such calls if 'this' is reassigned to another slot later from that targeted slot.

If I am reading the spec correctly, this is also perfectly legal. It only defines on what slot 'this' is passed to a method. I do neither think such reassignment is affecting the JIT where 'this' is no longer a special variable once the verifier is satisfied.

Furthermore, Byte Buddy retains the 'this' variable at slot 0. As a matter of fact, that's the point of it. It makes a defensive copy of the original array to function when obfuscators or alternative JVM languages are used which sometimes reassign slots that are no longer used in the code. Therefore, I am wondering what both IDEs are doing if this is both available at slot 0 and on the copied slot that is referrenced in the LVT. It would be available from both locations.

[raphw]

I added an exemption for remapping the this variable's index. As it is assumed immutable, that exemption works for the this variable only but it seems to yield the correct result in the debugger. All other variables are still remapped what is necessary since they might be mutated but the debugger seems to accept this.

In this context I noticed another unfortunate break point confusion. Since we prepend the line number to the very beginning of any method in order to add that line number to exceptions we throw from mocked methods, the debugger will stop if a break point is placed in the very first line of a method, even if the mocked path of the inline-mocked method is dispatched. In this case, no variables (including this) are displayed since the real code is never invoked but the call gets redirected into Mockito's own infrastructure. @TimvdLippe I think we rather have the Mockito exceptions show the right line number rather then not break in the first line of a method in case that a break point is set there, but maybe we should document this somewhere?

[TimvdLippe]

@raphw Yes I think that makes the most sense. We should probably add that caveat to https://javadoc.io/doc/org.mockito/mockito-core/latest/org/mockito/Mockito.html#39

[iamkarlson]

Is there any way to hot fix this issue locally?

[TimvdLippe]

@iamkarlson We first need a new release of ByteBuddy. Then we can roll that into Mockito and release Mockito as well. We can do that relatively quickly, so it's mostly blocked on the ByteBuddy release.

[jwb441]

@TimvdLippe would that fix the eclipse as well as IntelliJ ?

[raphw]

It should fix both. A new release is underway in this minute.

[TimvdLippe]

We will need to wait until 1.11.1 is on Maven Central before we can update our dependency in Mockito.

[TimvdLippe]

It seems like the ByteBuddy release is experiencing some issues: https://github.com/raphw/byte-buddy/runs/2713960314

[raphw]

On it, javadoc does not like something, not reproducable locally but I'll find out what it is.

[raphw]

Syncing to Central now and should be available throughout the night. We'll update Mockito soon but in the mean time it should be sufficient to simply bump the Byte Buddy dependency to 1.11.1.

[TimvdLippe]

I am waiting for #2312 to get merged (after review) and then we can upload the new version of Mockito to Maven Central.

[jwb441]

Syncing to Central now and should be available throughout the night. We'll update Mockito soon but in the mean time it should be sufficient to simply bump the Byte Buddy dependency to 1.11.1.

I tried upping Byte Buddy to 1.11.1 and still recreated the issue. Will try when the never version of Mokito is uploaded.

thanks again!

[TimvdLippe]

Mockito 3.11.0 is being pushed to Maven Central atm: https://github.com/mockito/mockito/actions/runs/903060692

[jwb441]

I just took 3.11.0 and reran the test and unfortunately I am still seeing the same issue. Let me know if there is anything else that I can add.

[TimvdLippe]

@jwb441 Could you double-check that you aren't accidentally using an older version of ByteBuddy via a different transitive dependency? I can retest this issue tomorrow.

[raphw]

I just tested it myself in IntelliJ, and it does now work as expected. There is one caveat, however. If you set the breakpoint in the first line of the method, it will:

a) not contain the this reference if stopped in the first line.
b) trigger the breakpoint for mocked method calls.

This is something we can hardly avoid, unfortunately. But we should certainly document it in the Mockito class.

[jwb441]

@TimvdLippe and @raphw - I missed that part in the discussion so went back and added a few more lines to the method in the spied class and I noticed a few things. If I put the break point in the method that I'm calling it never gets tripped. When I put it in the test and step into it I'm getting the error on line one (expected) and then my next step always jumps to the end of the method - I do get 'this' at that point though.

[ciancheng]

still got the same error. I am using mockito-core 4.0.0

[ebeltranro]

Hi, any progress on that? We have the same issue with mockito-inline 5.2.0.

[raphw]

I just tested and I cannot observe this problem. Are you sure that the method is non-static?

[ebeltranro]

Hi @raphw

We are using mockito-core 4.11 and mockito-inline 4.11 in Eclipse IDE. Our tests do not enter the invoked method, and if we force the entry with step into (F5), we get the error reported above:

com.sun.jdi.InternalException: Got error code in reply:35 occurred retrieving 'this' from stack frame.

Surprisingly, if we remove the mockito-inline dependency, these problems disappear and debugging works correctly.

Is there a bug in mockito-inline? Does it have debugging support?

Thank in advance.

[raphw]

We do not support older versions. Have you tried with the latest version of Mockito?

To me the exception looks like a bug in a different tool, so you might try updating your JVM, too.

[ebeltranro]

Hello again and thanks for your attention,

I confirm that with mockito-core 5.12 and modifying the MockerMock to subclass, my code is correctly debugged. But with the default MockMaker (inline) I can't debug, my internal breakpoints are ignored.

Can you please confirm if MockMaker inline is compatible with debug mode?

[raphw]

Where are you setting the breakpoint? You mean when invoking the actual method?

[ebeltranro]

Where are you setting the breakpoint? You mean when invoking the actual method?

The breakpoints that are causing me problems with MockMaker inline (I mean, the execution doesn't stop) are the ones that are in the real methods, invoked by the spied object.

[raphw]

If the method has multiple lines: does it also happen in a line further down?

[ebeltranro]

Update: I just tried the same thing with IntelliJ IDE instead of Eclipse and with IntelliJ all my breakpoints are working correctly.

Maybe it's easier for everyone to show you my code. I think it's very easy to reproduce. I have a test class SaludoTest.java, which creates a SaludoImpl spy object and then executes the method SaludoImpl.sayHello(). I want the code to stop in the sayHello() method itself.

I hope this helps. Thank you very much.

[raphw]

I also assume that is not related to the JVM but to the IDE not accepting instrumented code.