Segmentation fault when creating Fury instance

[wshtbla]

Search before asking

• I had searched in the issues and found no similar issues.

Version

Fury: 0.4.1
OS: CentOS Linux release 7.9.2009
JDK: Azul Zing JDK8 (zing21.08.1.0-1-jdk8.0.302-linux_x64)

Component(s)

Java

Minimal reproduce step

Start our Java web server, when running serialization, it will initiate Fury instance as ThreadLocal using ThreadSafeFury

Here's how we use Fury:

public class SerializationTranscoderV2<T> extends BaseTranscoder<T> {
    private static final Logger s_logger = Logger.getLogger(SerializationTranscoderV2.class);
    private static final ThreadSafeFury fury = Fury.builder()
            .withLanguage(Language.JAVA)
            .requireClassRegistration(false)
            .buildThreadSafeFury();

    @Override
    protected byte[] objectToBinary(T o) throws IOException {
        return fury.serialize(o);
    }

    @Override
    protected T binaryToObject(InputStream in) throws IOException {
        T object;
        try {
            byte[] data = IOUtils.toByteArray(in);
            object = (T) fury.deserialize(data);

        }
        catch (Exception e) {
            s_logger.error("Failed to read object from stream", e);
            object = null;
        }
        in.close();
        return object;
    }

What did you expect to see?

When creating Fury instance (with constructor in Fury.java which is called from ThreadLocalFury with ThreadSafeFury), it succeeds.

What did you see instead?

When creating Fury instance (with constructor in Fury.java which is called from ThreadLocalFury with ThreadSafeFury), it fails with segmentation fault.

Java server stdout:

2024-01-16 03:19:00,184 main INFO The nolookups option will be ignored. Message Lookups are no longer supported.
2024-01-16 03:19:00,187 main INFO The nolookups option will be ignored. Message Lookups are no longer supported.
2024-01-16 03:19:00,188 main INFO The nolookups option will be ignored. Message Lookups are no longer supported.
0: rip=0x00007f827ca7479b @rip=[0xffffffffffffffff] (os::print_hotspot_os_backtrace()+27) [gcc frame, calls gcc]
1: rip=0x00007f827c34c9dd @rip=[0x00007f82765ea918] (jvm_unexpected_exception_handler+157) [gcc frame, calls gcc]
2: rip=0x00007f827c34ca85 @rip=[0x00007f82765ea9c8] (jvm_unexpected_exception_helper+69) [gcc frame, calls gcc]
3: rip=0x00007f827b5f1767 @rip=[0x00007f82765ea9d8] (jvm_unexpected_exception_handler_wrapper()+615) [gcc frame, calls gcc]
4: rip=0x00007f827c191932 @rip=[0x00007f82765eaa38] (GPGC_Collector::mutator_relocate_object(oopDesc*)+50) [gcc frame, calls gcc]
5: rip=0x00007f827c269a15 @rip=[0x00007f82765eabd8] (oopDesc* GPGC_TLB::lvb_trap_from_c<(OopHelper::EncodingHint)0>(Thread*, oopDesc*, oopDesc* volatile*)+149) [gcc frame, calls gcc]
6: rip=0x00007f827c269750 @rip=[0x00007f82765eac68] (GPGC_TLB::lvb_trap_from_asm(Thread*, oopDesc*, oopDesc* volatile*)+144) [gcc frame, calls gcc]
7: rip=0x000000003000b3da @rip=[0x00007f82765eacb8] (lvb_handler+858) [java frame, calls java]
#
# An unexpected error has been detected by Java Runtime Environment:
#
# Segmentation fault (0xb) at pc=0x7f827c191932, pid=46777, tid=46781
#
# Java VM: Zing 64-Bit Tiered VM (1.8.0_302-zing_21.08.1.0-b1-product-linux-X86_64, mixed mode)
# Problematic frame:
# V [libjvm.so+0x2fd932] GPGC_Collector::mutator_relocate_object(oopDesc*)+0x32
#
# An error report file with more information is saved as:
# /opt/taboola/trc/hs_err_pid46777.log
#
# If you would like to submit a bug report, please visit:
# http://www.azul.com/support/

Zing error report:

Stack: [0x7f82763f0000,0x7f82765f0000], sp=0x7f82765eaa40, free space=2026k
Native frames: (J Jn Jo=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x2fd932] GPGC_Collector::mutator_relocate_object(oopDesc*)+0x32
V [libjvm.so+0x3d5a15] oopDesc* GPGC_TLB::lvb_trap_from_c<(OopHelper::EncodingHint)0>(Thread*, oopDesc*, oopDesc* volatile*)+0x95
V [libjvm.so+0x3d5750] GPGC_TLB::lvb_trap_from_asm(Thread*, oopDesc*, oopDesc* volatile*)+0x90
v ~StubRoutines::lvb_handler
Jn sun.misc.Unsafe.getObject(Ljava/lang/Object;J)Ljava/lang/Object;
j io.fury.util.unsafe._Lookup.<clinit>()V+24
v ~StubRoutines::call_stub
v ~StubRoutines::interpreter_resolve_invoke_stub
j io.fury.util.unsafe._JDKAccess$1.computeValue(Ljava/lang/Class;)Ljava/lang/invoke/MethodHandles$Lookup;+1
j io.fury.util.unsafe._JDKAccess$1.computeValue(Ljava/lang/Class;)Ljava/lang/Object;+2
j java.lang.ClassValue.getFromHashMap(Ljava/lang/Class;)Ljava/lang/Object;+29
j java.lang.ClassValue.getFromBackup([Ljava/lang/ClassValue$Entry;Ljava/lang/Class;)Ljava/lang/Object;+17
j java.lang.ClassValue.get(Ljava/lang/Class;)Ljava/lang/Object;+27
j io.fury.util.unsafe._JDKAccess._trustedLookup(Ljava/lang/Class;)Ljava/lang/invoke/MethodHandles$Lookup;+15
j io.fury.serializer.StringSerializer.<clinit>()V+159
v ~StubRoutines::call_stub
v ~StubRoutines::interpreter__new_stub
j io.fury.resolver.ClassResolver.addDefaultSerializers()V+4
j io.fury.resolver.ClassResolver.initialize()V+329
j io.fury.Fury.<init>(Lio/fury/config/FuryBuilder;Ljava/lang/ClassLoader;)V+131
j io.fury.config.FuryBuilder.newFury(Lio/fury/config/FuryBuilder;Ljava/lang/ClassLoader;)Lio/fury/Fury;+6
j io.fury.config.FuryBuilder.lambda$buildThreadLocalFury$0(Ljava/lang/ClassLoader;)Lio/fury/Fury;+2
j io.fury.config.FuryBuilder$$Lambda$lambda$buildThreadLocalFury$0$3997846765.apply(Ljava/lang/Object;)Ljava/lang/Object;+8
j io.fury.util.LoaderBinding.setClassLoader(Ljava/lang/ClassLoader;Lio/fury/util/LoaderBinding$StagingType;)V+116
j io.fury.util.LoaderBinding.setClassLoader(Ljava/lang/ClassLoader;)V+5
j io.fury.ThreadLocalFury.lambda$new$1(Ljava/util/function/Function;)Lio/fury/util/LoaderBinding;+16
j io.fury.ThreadLocalFury$$Lambda$lambda$new$1$2324057901.get()Ljava/lang/Object;+4
j java.lang.ThreadLocal$SuppliedThreadLocal.initialValue()Ljava/lang/Object;+4
j java.lang.ThreadLocal.setInitialValue()Ljava/lang/Object;+1

Can see it failed from
j io.fury.util.unsafe._JDKAccess$1.computeValue(Ljava/lang/Class;)Ljava/lang/invoke/MethodHandles$Lookup;+1

Anything Else?

We've also found a strange way to fix this, which is to run the Java server with Lightrun (It's a remote monitoring tool that uses bytecode instrumentation to capture log/metrics/callstack from a running Java program). We suspect there're some JIT/bytecode/instrumentation operations that will be allowed when running our server with Lightrun agent, and Fury might also require it.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[chaokunyang]

Hi @wshtbla , thanks for reporting this bug. Is there any reproduction code? It's not easy to trouble shooting a crash issue without running it locally

[wshtbla]

@chaokunyang I'll try to find one. It's a heavy and legacy server, so might not be easy. Will let you know.

[wshtbla]

We're checking with Azul as well, and good news is the new Zing version 23.08.300.0-2-jdk8.0.401-linux_x64 can work with Fury.
We're still checking why the old version fail with some JVM dump files. Will close this issue first.

[wshtbla]

@chaokunyang
The reason for the crush was because Fury uses Unsafe API that requires Zing to enable a JVM option called UseTrueObjectsForUnsafe. In old Zing, this option is disabled by default, but in new Zing, this option is enabled by default.

By the way, from Zing Engineer, they said Fury might have incorrect usage of Unsafe API.
I will copy their words here (see below):

---

Here is the code which caused the crash: https://github.com/apache/incubator-fury/blob/main/java/fury-core/src/main/java/org/apache/fury/util/unsafe/_Lookup.java#L44
I see incorrect usage of Unsafe API.

How it looks:

     try {
       Field implLookup = Lookup.class.getDeclaredField("IMPL_LOOKUP");
       long fieldOffset = _JDKAccess.UNSAFE.staticFieldOffset(implLookup);
       trustedLookup = (Lookup) _JDKAccess.UNSAFE.getObject(Lookup.class, fieldOffset);
     } catch (Throwable ignored) {
       // ignored
     }

How it should be.

     try {
       Field implLookup = Lookup.class.getDeclaredField("IMPL_LOOKUP");
       long fieldOffset = _JDKAccess.UNSAFE.staticFieldOffset(implLookup);
       Object fieldBase = _JDKAccess.UNSAFE.staticFieldBase(implLookup);
       trustedLookup = (Lookup) _JDKAccess.UNSAFE.getObject(fieldBase, fieldOffset);
     } catch (Throwable ignored) {
       // ignored
     }

When -UseTrueObjectsForUnsafe is used, UNSAFE.getObject expects raw klassOop, not java.lang.Class instance. It leads to issue.
This code kind of is ok when we use +UseTrueObjectsForUnsafe and OpenJDK, because 'staticFieldBase' will return Lookup.class in this case. But this is not guaranteed.

Please check official documentation, how to get input parameters for Unsafe operations:
https://github.com/keerath/openjdk-8-source/blob/master/jdk/src/share/classes/sun/misc/Unsafe.java#L114

---

Does this make sense?

[chaokunyang]

Hi @wshtbla , thanks so much for sharing this to us. We did use the wrong Unsafe API for getting static field. This is not an easy bug to locate, thanks again for sharing the fix to us. I'll submit a fix right now.