TString.scalarOf core dumps on OSX

[jakeri]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): OSX 11.3.1 (intel)
• TensorFlow installed from (source or binary): binary TF Java 0.3.1 and 0.4.0-SNAPSHOT
• TensorFlow version (use command below): 2.4.1 as built with TF Java 0.3.1
• CUDA/cuDNN version: No
• GPU model and memory: No

Describe the current behavior

Calling TString.scalarOf("hello"); will core dump on OSX. It will actually succeed 1 out of 10 times.

Describe the expected behavior

I want to be able to create a TString without a core dump.

Code to reproduce the issue

JUnit test

    @Test
    public void testTString() throws Exception {
        TString.scalarOf("hello");
    }

Other info / logs

Running same test case through docker linux vm succeeds.

One core dump from a run:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00000001422d724c, pid=16660, tid=5635
#
# JRE version: OpenJDK Runtime Environment (11.0.2+9) (build 11.0.2+9)
# Java VM: OpenJDK 64-Bit Server VM (11.0.2+9, mixed mode, tiered, compressed oops, g1 gc, bsd-amd64)
# Problematic frame:
# C  [libtensorflow_cc.2.dylib+0x822924c]  tensorflow::TensorInterface::ByteSize() const+0xc
#
# Core dump will be written. Default location: /cores/core.16660
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  S U M M A R Y ------------

Command Line: /Users/jakob/proj/tfjava031/target/surefire/surefirebooter11868128427691651602.jar /Users/jakob/proj/tfjava031/target/surefire/surefire16140580774728156639tmp /Users/jakob/proj/tfjava031/target/surefire/surefire_012592081837142384420tmp

Host: MacBookPro13,2 x86_64 3100 MHz, 4 cores, 16G, Darwin 20.4.0
Time: Tue May 11 11:47:43 2021 CEST elapsed time: 3 seconds (0d 0h 0m 3s)

---------------  T H R E A D  ---------------

Current thread (0x00007fb48f809000):  JavaThread "main" [_thread_in_native, id=5635, stack(0x00007000053f2000,0x00007000054f2000)]

Stack: [0x00007000053f2000,0x00007000054f2000],  sp=0x00007000054eeea8,  free space=1011k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libtensorflow_cc.2.dylib+0x822924c]  tensorflow::TensorInterface::ByteSize() const+0xc
j  org.tensorflow.internal.buffer.TensorBuffers.tensorMemory(Lorg/tensorflow/internal/c_api/TF_Tensor;)Lorg/bytedeco/javacpp/Pointer;+5
j  org.tensorflow.internal.buffer.TensorBuffers.toStrings(Lorg/tensorflow/internal/c_api/TF_Tensor;J)Lorg/tensorflow/internal/buffer/ByteSequenceTensorBuffer;+1
j  org.tensorflow.internal.types.TStringMapper.mapDense(Lorg/tensorflow/RawTensor;)Lorg/tensorflow/types/TString;+11
j  org.tensorflow.internal.types.TStringMapper.mapDense(Lorg/tensorflow/RawTensor;)Lorg/tensorflow/types/family/TType;+2
j  org.tensorflow.RawTensor.asTypedTensor()Lorg/tensorflow/types/family/TType;+8
j  org.tensorflow.Tensor.of(Ljava/lang/Class;Lorg/tensorflow/ndarray/Shape;J)Lorg/tensorflow/types/family/TType;+10
j  org.tensorflow.Tensor.of(Ljava/lang/Class;Lorg/tensorflow/ndarray/Shape;JLjava/util/function/Consumer;)Lorg/tensorflow/types/family/TType;+3
j  org.tensorflow.types.TString.tensorOf(Ljava/nio/charset/Charset;Lorg/tensorflow/ndarray/NdArray;)Lorg/tensorflow/types/TString;+28
j  org.tensorflow.types.TString.tensorOf(Lorg/tensorflow/ndarray/NdArray;)Lorg/tensorflow/types/TString;+4
j  org.tensorflow.types.TString.scalarOf(Ljava/lang/String;)Lorg/tensorflow/types/TString;+4
j  com.peltarion.tf.test.JakobTest.testTString()V+2
v  ~StubRoutines::call_stub
V  [libjvm.dylib+0x39b736]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x21a
V  [libjvm.dylib+0x64c917]  invoke(InstanceKlass*, methodHandle const&, Handle, bool, objArrayHandle, BasicType, objArrayHandle, bool, Thread*)+0x4e6
V  [libjvm.dylib+0x64c3e7]  Reflection::invoke_method(oopDesc*, Handle, objArrayHandle, Thread*)+0xf5
V  [libjvm.dylib+0x42236d]  JVM_InvokeMethod+0x20a
...

[Craigacp]

When running this on macOS via jshell (using a with dependencies jar made by another project which contains v0.3.1) I get this stack trace in the hotspot dump:

Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libtensorflow_cc.2.dylib+0x8228fb7]  TF_TensorData+0x7
C  [libjnitensorflow.dylib+0x360e4]  Java_org_tensorflow_internal_c_1api_global_tensorflow_TF_1TensorData+0x34
j  org.tensorflow.internal.c_api.global.tensorflow.TF_TensorData(Lorg/tensorflow/internal/c_api/TF_Tensor;)Lorg/bytedeco/javacpp/Pointer;+0
j  org.tensorflow.internal.buffer.TensorBuffers.tensorMemory(Lorg/tensorflow/internal/c_api/TF_Tensor;)Lorg/bytedeco/javacpp/Pointer;+1
j  org.tensorflow.internal.buffer.TensorBuffers.toStrings(Lorg/tensorflow/internal/c_api/TF_Tensor;J)Lorg/tensorflow/internal/buffer/ByteSequenceTensorBuffer;+1
j  org.tensorflow.internal.types.TStringMapper.mapDense(Lorg/tensorflow/RawTensor;)Lorg/tensorflow/types/TString;+11
j  org.tensorflow.internal.types.TStringMapper.mapDense(Lorg/tensorflow/RawTensor;)Lorg/tensorflow/types/family/TType;+2
j  org.tensorflow.RawTensor.asTypedTensor()Lorg/tensorflow/types/family/TType;+8
j  org.tensorflow.Tensor.of(Ljava/lang/Class;Lorg/tensorflow/ndarray/Shape;J)Lorg/tensorflow/types/family/TType;+10
j  org.tensorflow.Tensor.of(Ljava/lang/Class;Lorg/tensorflow/ndarray/Shape;JLjava/util/function/Consumer;)Lorg/tensorflow/types/family/TType;+3
j  org.tensorflow.types.TString.tensorOf(Ljava/nio/charset/Charset;Lorg/tensorflow/ndarray/NdArray;)Lorg/tensorflow/types/TString;+28
j  org.tensorflow.types.TString.tensorOf(Lorg/tensorflow/ndarray/NdArray;)Lorg/tensorflow/types/TString;+4
j  org.tensorflow.types.TString.scalarOf(Ljava/lang/String;)Lorg/tensorflow/types/TString;+4

Can you check that you've got the same stack trace?

Also for the record this test passes when it's run inside tensorflow-core-api's TString test class, which is very odd.

[jakeri]

I realised that forgot to add a stack trace. But is added.
They are actually slightly different. I've captured three below.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00000001422d724c, pid=16660, tid=5635
#
# JRE version: OpenJDK Runtime Environment (11.0.2+9) (build 11.0.2+9)
# Java VM: OpenJDK 64-Bit Server VM (11.0.2+9, mixed mode, tiered, compressed oops, g1 gc, bsd-amd64)
# Problematic frame:
# C  [libtensorflow_cc.2.dylib+0x822924c]  tensorflow::TensorInterface::ByteSize() const+0xc
#
# Core dump will be written. Default location: /cores/core.16660
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  S U M M A R Y ------------

Command Line: /Users/jakob/proj/tfjava031/target/surefire/surefirebooter11868128427691651602.jar /Users/jakob/proj/tfjava031/target/surefire/surefire16140580774728156639tmp /Users/jakob/proj/tfjava031/target/surefire/surefire_012592081837142384420tmp

Host: MacBookPro13,2 x86_64 3100 MHz, 4 cores, 16G, Darwin 20.4.0
Time: Tue May 11 11:47:43 2021 CEST elapsed time: 3 seconds (0d 0h 0m 3s)

---------------  T H R E A D  ---------------

Current thread (0x00007fb48f809000):  JavaThread "main" [_thread_in_native, id=5635, stack(0x00007000053f2000,0x00007000054f2000)]

Stack: [0x00007000053f2000,0x00007000054f2000],  sp=0x00007000054eeea8,  free space=1011k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libtensorflow_cc.2.dylib+0x822924c]  tensorflow::TensorInterface::ByteSize() const+0xc
j  org.tensorflow.internal.buffer.TensorBuffers.tensorMemory(Lorg/tensorflow/internal/c_api/TF_Tensor;)Lorg/bytedeco/javacpp/Pointer;+5
j  org.tensorflow.internal.buffer.TensorBuffers.toStrings(Lorg/tensorflow/internal/c_api/TF_Tensor;J)Lorg/tensorflow/internal/buffer/ByteSequenceTensorBuffer;+1
j  org.tensorflow.internal.types.TStringMapper.mapDense(Lorg/tensorflow/RawTensor;)Lorg/tensorflow/types/TString;+11
j  org.tensorflow.internal.types.TStringMapper.mapDense(Lorg/tensorflow/RawTensor;)Lorg/tensorflow/types/family/TType;+2
j  org.tensorflow.RawTensor.asTypedTensor()Lorg/tensorflow/types/family/TType;+8
j  org.tensorflow.Tensor.of(Ljava/lang/Class;Lorg/tensorflow/ndarray/Shape;J)Lorg/tensorflow/types/family/TType;+10
j  org.tensorflow.Tensor.of(Ljava/lang/Class;Lorg/tensorflow/ndarray/Shape;JLjava/util/function/Consumer;)Lorg/tensorflow/types/family/TType;+3
j  org.tensorflow.types.TString.tensorOf(Ljava/nio/charset/Charset;Lorg/tensorflow/ndarray/NdArray;)Lorg/tensorflow/types/TString;+28
j  org.tensorflow.types.TString.tensorOf(Lorg/tensorflow/ndarray/NdArray;)Lorg/tensorflow/types/TString;+4
j  org.tensorflow.types.TString.scalarOf(Ljava/lang/String;)Lorg/tensorflow/types/TString;+4
j  com.peltarion.tf.test.JakobTest.testTString()V+2
v  ~StubRoutines::call_stub
V  [libjvm.dylib+0x39b736]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x21a
V  [libjvm.dylib+0x64c917]  invoke(InstanceKlass*, methodHandle const&, Handle, bool, objArrayHandle, BasicType, objArrayHandle, bool, Thread*)+0x4e6
V  [libjvm.dylib+0x64c3e7]  Reflection::invoke_method(oopDesc*, Handle, objArrayHandle, Thread*)+0xf5
V  [libjvm.dylib+0x42236d]  JVM_InvokeMethod+0x20a

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000000013c6ee24c, pid=21898, tid=5635
#
# JRE version: OpenJDK Runtime Environment (11.0.2+9) (build 11.0.2+9)
# Java VM: OpenJDK 64-Bit Server VM (11.0.2+9, mixed mode, tiered, compressed oops, g1 gc, bsd-amd64)
# Problematic frame:
# C  [libtensorflow_cc.2.dylib+0x822924c]  _ZNK10tensorflow15TensorInterface8ByteSizeEv+0xc
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  S U M M A R Y ------------

Command Line: -ea -Dfile.encoding=UTF-8 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner -version 3 -port 49881 -testLoaderClass org.eclipse.jdt.internal.junit5.runner.JUnit5TestLoader -loaderpluginname org.eclipse.jdt.junit5.runtime -test com.peltarion.tf.test.JakobTest:testTString

Host: MacBookPro13,2 x86_64 3100 MHz, 4 cores, 16G, Darwin 20.4.0
Time: Tue May 11 15:34:42 2021 CEST elapsed time: 9 seconds (0d 0h 0m 9s)

---------------  T H R E A D  ---------------

Current thread (0x00007f8ed4809800):  JavaThread "main" [_thread_in_native, id=5635, stack(0x000070000a197000,0x000070000a297000)]

Stack: [0x000070000a197000,0x000070000a297000],  sp=0x000070000a293c88,  free space=1011k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libtensorflow_cc.2.dylib+0x822924c]  _ZNK10tensorflow15TensorInterface8ByteSizeEv+0xc
j  org.tensorflow.internal.buffer.TensorBuffers.tensorMemory(Lorg/tensorflow/internal/c_api/TF_Tensor;)Lorg/bytedeco/javacpp/Pointer;+5
j  org.tensorflow.internal.buffer.TensorBuffers.toStrings(Lorg/tensorflow/internal/c_api/TF_Tensor;J)Lorg/tensorflow/internal/buffer/ByteSequenceTensorBuffer;+1
j  org.tensorflow.internal.types.TStringMapper.mapDense(Lorg/tensorflow/RawTensor;)Lorg/tensorflow/types/TString;+11
j  org.tensorflow.internal.types.TStringMapper.mapDense(Lorg/tensorflow/RawTensor;)Lorg/tensorflow/types/family/TType;+2
j  org.tensorflow.RawTensor.asTypedTensor()Lorg/tensorflow/types/family/TType;+8
j  org.tensorflow.Tensor.of(Ljava/lang/Class;Lorg/tensorflow/ndarray/Shape;J)Lorg/tensorflow/types/family/TType;+10
j  org.tensorflow.Tensor.of(Ljava/lang/Class;Lorg/tensorflow/ndarray/Shape;JLjava/util/function/Consumer;)Lorg/tensorflow/types/family/TType;+3
j  org.tensorflow.types.TString.tensorOf(Ljava/nio/charset/Charset;Lorg/tensorflow/ndarray/NdArray;)Lorg/tensorflow/types/TString;+28
j  org.tensorflow.types.TString.tensorOf(Lorg/tensorflow/ndarray/NdArray;)Lorg/tensorflow/types/TString;+4
j  org.tensorflow.types.TString.scalarOf(Ljava/lang/String;)Lorg/tensorflow/types/TString;+4
j  com.peltarion.tf.test.JakobTest.testTString()V+2

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00000001384bd24c, pid=21915, tid=9987
#
# JRE version: OpenJDK Runtime Environment (11.0.2+9) (build 11.0.2+9)
# Java VM: OpenJDK 64-Bit Server VM (11.0.2+9, mixed mode, tiered, compressed oops, g1 gc, bsd-amd64)
# Problematic frame:
# C  [libtensorflow_cc.2.dylib+0x822924c]  tensorflow::TensorInterface::ByteSize() const+0xc
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  S U M M A R Y ------------

Command Line: -ea -Dfile.encoding=UTF-8 org.eclipse.jdt.internal.junit.runner.RemoteTestRunner -version 3 -port 50178 -testLoaderClass org.eclipse.jdt.internal.junit5.runner.JUnit5TestLoader -loaderpluginname org.eclipse.jdt.junit5.runtime -test com.peltarion.tf.test.JakobTest:testTString

Host: MacBookPro13,2 x86_64 3100 MHz, 4 cores, 16G, Darwin 20.4.0
Time: Tue May 11 15:35:53 2021 CEST elapsed time: 7 seconds (0d 0h 0m 7s)

---------------  T H R E A D  ---------------

Current thread (0x00007fc912019800):  JavaThread "main" [_thread_in_native, id=9987, stack(0x0000700005453000,0x0000700005553000)]

Stack: [0x0000700005453000,0x0000700005553000],  sp=0x000070000554fc88,  free space=1011k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libtensorflow_cc.2.dylib+0x822924c]  tensorflow::TensorInterface::ByteSize() const+0xc
j  org.tensorflow.internal.buffer.TensorBuffers.tensorMemory(Lorg/tensorflow/internal/c_api/TF_Tensor;)Lorg/bytedeco/javacpp/Pointer;+5
j  org.tensorflow.internal.buffer.TensorBuffers.toStrings(Lorg/tensorflow/internal/c_api/TF_Tensor;J)Lorg/tensorflow/internal/buffer/ByteSequenceTensorBuffer;+1
j  org.tensorflow.internal.types.TStringMapper.mapDense(Lorg/tensorflow/RawTensor;)Lorg/tensorflow/types/TString;+11
j  org.tensorflow.internal.types.TStringMapper.mapDense(Lorg/tensorflow/RawTensor;)Lorg/tensorflow/types/family/TType;+2
j  org.tensorflow.RawTensor.asTypedTensor()Lorg/tensorflow/types/family/TType;+8
j  org.tensorflow.Tensor.of(Ljava/lang/Class;Lorg/tensorflow/ndarray/Shape;J)Lorg/tensorflow/types/family/TType;+10
j  org.tensorflow.Tensor.of(Ljava/lang/Class;Lorg/tensorflow/ndarray/Shape;JLjava/util/function/Consumer;)Lorg/tensorflow/types/family/TType;+3
j  org.tensorflow.types.TString.tensorOf(Ljava/nio/charset/Charset;Lorg/tensorflow/ndarray/NdArray;)Lorg/tensorflow/types/TString;+28
j  org.tensorflow.types.TString.tensorOf(Lorg/tensorflow/ndarray/NdArray;)Lorg/tensorflow/types/TString;+4
j  org.tensorflow.types.TString.scalarOf(Ljava/lang/String;)Lorg/tensorflow/types/TString;+4
j  com.peltarion.tf.test.JakobTest.testTString()V+2

[jakeri]

It could be related to first start. Because:

Running TFloat32 creation never fails.

    @Test
    public void testTFloat32() throws Exception {
        TFloat32.vectorOf(2.3f);
    }

And if I run a creation of TFloat32 before creating a TString it will also never fail.

    @Test
    public void testTs() throws Exception {
        TFloat32.vectorOf(2.3f);
        TString.scalarOf("hello");
    }

So OSX with a TString creation as first touch point with Tensorflow Java on a newly started JVM will core dump.

[karllessard]

The failure happens when we try to retrieve the size of the tensor in byte, when calling TF_TensorByteSize here.

Somehow, it looks like the buffer of the allocated tensor is null in the native code. I'll continue the investigation a bit later.

[karllessard]

Ok, went a bit further in this and it is kind of scary, looks like the C API behaves differently when we observe it than when we don't, like the observer effect in quantum physics (I didn't know TF was that scientifically advanced!)

So in that particular line, we compute the size of a C API String Tensor object (TF_TString) to determine the size of the buffer to allocate.

If a float tensor has been created before the string one, the returned value is 24 (good). Otherwise, it is 8 (bad). But if simply I add a breakpoint to that line and invoke Loader.sizeOf(TF_TString.class) in my debugger console, without the float tensor, then I get 24 again and the rest of the execution succeeds. So why my debugger console and the normal execution receives a different value?

@saudet , any idea why Loader.sizeOf could return different values depending on how or when it is invoked?

[saudet]

That could in theory be caused by the WeakHashMap not getting properly initialized before we query it, but I've never encountered an issue like that. Everything gets called from JNI, with synchronization going on in the class loader and all, so unless we try to query it before returning from Loader.load(), I don't see how this could happen. Besides, the crash happens with TF Java 0.2.0 as well where we are not calling Loader.sizeof(). To make really sure this isn't happening because of something related to thread-safety issues with WeakHashMap, try to make Loader.sizeof() synchronized here:
https://github.com/bytedeco/javacpp/blob/master/src/main/java/org/bytedeco/javacpp/Loader.java#L1945

However, I don't see how that would change the value being returned. It's just a constant on the native size. If the above doesn't help, the other thing that could be happening is memory being already corrupted at that point. If this were on Linux, I'd say we could try to run Valgrind on this, but with Mac it's full of weird bugs, it's not going to help, I think, but maybe it's better these days. Anyway, if the problem is memory corruption, it's happening before that. This is just a symptom, not the cause.

[karllessard]

So that was it... If the first thing we do using TF Java is to create a string tensor, we fall in a particular path where Loader.load() will not be invoked before (which ultimately comes by loading the TensorFlow class) and hence the size of TF_TString is unknown (in all other cases, we hit a path where the native library is loaded before we have anything to do with the C API bindings). I still don't know why my debugger console was not having this issue but anyway, let's ignore that.

So just calling TensorFlow.version() before creating the tensor works. I'll check where would be the best location to add this and make a PR for it, thanks