Snappy causes SIGBUS jvm crasher on HP-UX via UnsafeMemory.loadInt()

[ekeithkw]

Steps to reproduce:

1. Install HP-UX Java 7 (e.g. from “Itanium_JDK_JRE_7.0.09_-_Feb_2014_Z7550-01326_java_17009_ia.depot”) to a local folder.
2. Set JDK_HOME to the above folder.
3. Download CLion for Linux, from jetbrains.com > Products > CLion > Early Access Program (EAP); gunzip and “tar xvf” to unpack (e.g. to /opt).
4. Run “/opt/clion-140.569.17/bin/clion.sh”. Deselect all plug-ins.
5. It appears necessary to create or open a project (failure occurs when opening a project).

Snippet from hs_err.log file:
Current thread (01370200): JavaThread "ApplicationImpl pooled thread 3" [_thread_in_vm, id=30, lwp_id=4383651, stack(2d601000,2d641000)]

siginfo:si_signo=SIGBUS: si_errno=0, si_code=1 (BUS_ADRALN), si_addr=2f4b204d

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j sun.misc.Unsafe.getInt(Ljava/lang/Object;J)I+0
j org.iq80.snappy.UnsafeMemory.loadInt([BI)I+50
j org.iq80.snappy.SnappyInternalUtils.loadInt([BI)I+5
j org.iq80.snappy.SnappyCompressor.findCandidate([BIIII[SI)[I+19
j org.iq80.snappy.SnappyCompressor.compressFragment([BII[BI[S)I+157
j org.iq80.snappy.SnappyCompressor.compress([BII[BI)I+63
j org.iq80.snappy.Snappy.compress([BII[BI)I+6
j org.iq80.snappy.SnappyOutputStream.writeCompressed([BII)V+27
j org.iq80.snappy.SnappyOutputStream.flushBuffer()V+17
j org.iq80.snappy.SnappyOutputStream.flush()V+18
j sun.nio.cs.StreamEncoder.implFlush()V+15
j sun.nio.cs.StreamEncoder.flush()V+12
j java.io.OutputStreamWriter.flush()V+4
j org.jdom.output.XMLOutputter.output(Lorg/jdom/Element;Ljava/io/Writer;)V+12
j com.intellij.openapi.components.impl.stores.StateMap.a(Lorg/jdom/Element;)[B+85
j com.intellij.openapi.components.impl.stores.StateMap.getStateAndArchive(Ljava/lang/String;)Lorg/jdom/Element;+68

HP has kindly provided some analysis:

In our internal discussions, we have arrived at the following conclusions (by understanding the code of snappy):
https://github.com/dain/snappy/blob/master/src/main/java/org/iq80/snappy/SnappyOutputStream.java & http://codenav.org/code.html?project=/org/iq80/snappy/snappy/0.1 .
In the call stack that we have seen:
#31 0x2000000073405c00 in interpreted frame: org.iq80.snappy.SnappyOutputStream::flushBuffer () ->void bci: 17

Provides the request to write buffer. The buffer in SnappyOutputStream.java is as follows:
private final byte[] buffer;

This buffer is passed between all the methods until it reaches:
#20
#21 0x60000000d56701e0:0 in Unsafe_GetInt () at /wsp/jinteg/SVN/jinteg_h7.0.11.rc1b1/hotspot/src/share/vm/prims/unsafe.cpp:396
#22 0x2000000073400d70 in Java native_call_stub frame
#23 0x2000000073413b70 in JNI frame: sun.misc.Unsafe::getInt (java.lang.Object, long) ->int
#24 0x2000000073405aa0 in interpreted frame: org.iq80.snappy.UnsafeMemory::loadInt (byte[], int) ->int bci: 50
#25 0x2000000073405de0 in interpreted frame: org.iq80.snappy.SnappyInternalUtils::loadInt (byte[], int) ->int bci: 5

Causing the exception (which you have observed). In the Java API code in file jdk/src/share/classes/sun/misc/Unsafe.java (available for your reference through OpenJDK) contains the following comments before method getInt():

    // These work on object fields in the Java heap.
    // They will not work on elements of packed arrays.
   Fetches a value from a given Java variable.
   More specifically, fetches a field or array element within the given
   object <code>o</code> at the given offset, or (if <code>o</code> is
   null) from the memory address whose numerical value is the given
   offset.
   <p>
   The results are undefined unless one of the following cases is true:
   <ul>
   <li>The offset was obtained from {@link #objectFieldOffset} on
   the {@link java.lang.reflect.Field} of some Java field and the object
   referred to by <code>o</code> is of a class compatible with that
   field's class.

   <li>The offset and object reference <code>o</code> (either null or
   non-null) were both obtained via {@link #staticFieldOffset}
   and {@link #staticFieldBase} (respectively) from the
   reflective {@link Field} representation of some Java field.

   <li>The object referred to by <code>o</code> is an array, and the offset
   is an integer of the form <code>B+N*S</code>, where <code>N</code> is
   a valid index into the array, and <code>B</code> and <code>S</code> are
   the values obtained by {@link #arrayBaseOffset} and {@link
   #arrayIndexScale} (respectively) from the array's class.  The value
   referred to is the <code>N</code><em>th</em> element of the array.

   </ul>
   <p>
   If one of the above cases is true, the call references a specific Java
   variable (field or array element). However, the results are undefined
   if that variable is not in fact of the type returned by this method.
   <p>
'''

The code in snappy calls getInt with an arbitrary offset, not obtained by any of the methods described above (apparently the only available documentation for this). The behavior for such invocation of sun.misc.Unsafe::getInt() is undefined regardless of the platform it is invoked in.


Note also that HP-UX Itanium is a big-endian architecture. See:
https://github.com/dain/snappy/commit/a7c9c3cdbd6a0d997975ccbde5c946e07c1684e2 

This patch may not yet be in the snappy.jar bundled with JetBrains' CLion, and may work around the issue.

[electrum]

I believe the first part of the analysis "The code in snappy calls getInt with an arbitrary offset, not obtained by any of the methods described above" is partially incorrect. The following comes from UnsafeMemory:

    private static final long BYTE_ARRAY_OFFSET = unsafe.arrayBaseOffset(byte[].class);

    @Override
    public int loadInt(byte[] data, int index)
    {
        assert index >= 0;
        assert index + 4 <= data.length;
        return unsafe.getInt(data, BYTE_ARRAY_OFFSET + index);
    }

Thus, the offset is not arbitrary: it is obtained via the arrayBaseOffset. A potential issue is that the index is NOT multiplied by the array scale obtained by arrayIndexScale, likely due to the assumption that a byte array scale is always one (which is the case for every JVM I have seen).

The likely due to the architecture being big-endian. However, I have filed an issue #25 for the byte array index scale issue.

Thanks for reporting this!