native-image -H:Optimize=2 leads to wrong execution / code.

[michael-simons]

Hello everyone.
My name is Michael, I work for Neo4j and I contributed the Neo4j extension to the Quarkus project. That extension worked nicely in native mode and stopped doing so since Quarkus upgraded to GraalVM 19.3 (See related issue here quarkusio/quarkus#6115).

A little background to our driver: We shade Netty and have some substitutions in place to make SSL and some reflections work, which you can find here:

• https://github.com/neo4j/neo4j-java-driver/tree/4.0.0/driver/src/main/resources/META-INF/native-image/org.neo4j/driver
• https://github.com/neo4j/neo4j-java-driver/tree/4.0.0/driver/src/main/java/org/neo4j/driver/internal/svm

The bug happens when the driver receives a message from the server in a pack stream format. I have a fully runnable example here https://github.com/michael-simons/neo4j-java-driver-native-example, but you need to have a running Neo4j instance to make use of it.

I was however able to pin point our issue to the code inside Unpacker, which you find here: https://github.com/neo4j/neo4j-java-driver/blob/4.0.0/driver/src/main/java/org/neo4j/driver/internal/packstream/PackStream.java#L401

In GraalVM prior to 19.3, regardless of -H:Optimize=n settings, the GraalVM native image compiler produced working code. Since 19.3, the various methods, and I think the partial switch statements, seem to fold into one which lead to wrong exceptions.

You'll find attached a reproducer, that doesn't need a running Neo4j instance. It's only dependency is io.netty:netty-buffer and I create the scenario when the message handler receives a welcome message from our server. The class optimizationissue.Reproducer creates a Netty ByteBuf from the message that the server sent and decodes it as the original Neo4j driver would do. It uses a copy of PackStream$Unpacker that I linked above.

The message starts with a struct header, so we start by calling optimizationissue.PackStream.Unpacker#unpackStructHeader. And while the message fit's a struct header, some other branch seems to be called.

The expected output is {server=Neo4j/4.0.0-rc01, connection_id=bolt-536}.

If you add -H:Optimize=0 to the native image generation, things work as expected.

Find the complete project attached and thanks for looking into this.

graal-issue.zip

graal-issue-no-netty-dependencies.zip

[eginez]

Hi @michael-simons is this a problem with jdk8 or jdk11?

[michael-simons]

The reproducer above was created and verified to fail on macOS 10.13 with GraalVM 19.3 CE JDK 8.

19.2 and prior it works as expected.

The people in the linked Quarkus Ticket using the Neo4j driver are probably on Linux or at least Quarkus’ CI is.

[michael-simons]

Just mvn clean package && ./target/reproducer should be enough to end with a stack trace and not with the expected exception.

[michael-simons]

Hi @eginez I can confirm that this happens on

• macOS Graal 19.3 JDK 11
• Linux amd64 Graal 19.3 JDK 8 and JDK 11

as well.

I also wanted to exclude the possibility that our usage of Netty leads to the error, so I created a second reproducer without Netty as dependency. This is attached to the original comment as https://github.com/oracle/graal/files/3972548/graal-issue-no-netty-dependencies.zip. Hope that helps. Thank you!

[eginez]

@michael-simons yes I was able to replicate the problem even in the head of master(building from source). We are working on it

[eginez]

@michael-simons we have found the source of the bug, we are in the process of running verifications on the fix. A commit with it will be merged shortly

[eginez]

@michael-simons the fix just landed in the master branch 276ff2b

Feel free to verify it against the downstream dependencies

[michael-simons]

I’ll gladly do this. Do you have snapshot builds or should I build GraalVM myself?

[eginez]

Fastest option is to build from source, until we have a build with it

[michael-simons]

I can confirm that the Neo4j driver works correctly again with 19.3.0.2. Thanks so much for making this a priority.