mvn install: Failed to execute goal org.bytedeco:javacpp:1.5.3:build

[aday00]

Thanks for making this Java API to Tensorflow! Would you kindly help me understand this error from mvn install:

[INFO] Reactor Summary for TensorFlow Java Parent 0.2.0-SNAPSHOT:
[INFO] 
[INFO] TensorFlow Java Parent ............................. SUCCESS [  1.774 s]
[INFO] TensorFlow NdArray Library ......................... SUCCESS [02:47 min]
[INFO] TensorFlow Core Parent ............................. SUCCESS [  0.187 s]
[INFO] TensorFlow Core Annotation Processor ............... SUCCESS [  3.157 s]
[INFO] TensorFlow Core API Library ........................ FAILURE [05:15 min]
[INFO] TensorFlow Core API Library Platform ............... SKIPPED
[INFO] TensorFlow Framework Library ....................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  08:09 min
[INFO] Finished at: 2020-08-01T04:56:29Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.bytedeco:javacpp:1.5.3:build (javacpp-build) on project tensorflow-core-api: Execution javacpp-build of goal org.bytedeco:javacpp:1.5.3:build failed: Process exited with an error: 132 -> [Help 1]utionException: Failed to execute goal org.bytedeco:javacpp:1.5.3:build (javacpp-build) on project tensorflow-core-api: Execution javacpp-build of goal org.bytedeco:javacpp:1.5.3:build failed: Process exited with an error: 132r.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution javacpp-build of goal org.bytedeco:javacpp:1.5.3:build failed: Process exited with an error: 132
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:148)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: java.lang.RuntimeException: Process exited with an error: 132
    at org.bytedeco.javacpp.tools.Builder.build (Builder.java:1026)
    at org.bytedeco.javacpp.tools.BuildMojo.execute (BuildMojo.java:411)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:566)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[ERROR] 
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]

My 'uname -a' is 'Linux 4.9.0-7-amd64 #1 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x86_64 GNU/Linux'. I'm running in Docker, w/ Java 11. Happy to use Java 8 instead if that helps.

I run CPU-only. Old CPU. No AVX. CPU flags are (from /proc/cpuinfo):

flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes rdrand lahf_lm 3dnowprefetch epb kaiser tpr_shadow vnmi flexpriority ept vpid tsc_adjust smep erms dtherm ida arat

I have a custom .whl file for my Tensorflow build from source, which excludes AVX instructions. However, tensorflow-java's 'mvn install' seems to build its own Tensorflow, and I'm not sure AVX is excluded. Moreover, my /var/log/messages shows an invalid opcode error during 'mvn install', so I suspect AVX issues etc:

Aug  1 00:56:28 kernel: [6829864.802276] traps: java_op_generat[5990] trap invalid opcode ip:7fe3188f1a40 sp:7ffe8f916cb8 error:0
Aug  1 00:56:28 kernel: [6829864.802289]  in libtensorflow_framework.so.2.2.0[7fe317356000+1ab9000]

I'd prefer to tell tensorflow-java to use the Tensorflow w/o AVX that I built and installed, rather than use tensorflow-java's mvn-install-built Java Parent Tensorflow. Is this possible? Please let me know if this is a reasonable fix, or what I should do. Not sure how to interpret the error messages. Thanks for your time and insights!

[aday00]

Looking a bit earlier in the mvn install -e -X output, we see an illegal instruction [AVX?] and core dump in the line /tmp/docker-share/tensorflow-java/tensorflow-core/tensorflow-core-api/build.sh: line 80: 31711 Illegal instruction (core dumped) $BAZEL_BIN/java_op_generator --output_dir=$GEN_SRCS_DIR --api_dirs=$BAZEL_SRCS/external/org_tensorflow/tensorflow/core/api_def/base_api,src/bazel/api_def $TENSORFLOW_LIB

[1 / 3,582] checking cached actions
INFO: Elapsed time: 320.642s, Critical Path: 18.46s
[6,563 / 6,563] checking cached actions
INFO: 0 processes.
[6,563 / 6,563] checking cached actions
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
Listing /tmp/docker-share/tensorflow-java/tensorflow-core/tensorflow-core-api/bazel-bin/external/org_tensorflow/tensorflow:
total 215244
drwxr-xr-x  5 root root      4096 Jul 31 09:20 c
drwxr-xr-x  5 root root      4096 Jul 31 17:03 cc
drwxr-xr-x  6 root root      4096 Jul 31 15:58 compiler
drwxr-xr-x 18 root root      4096 Jul 31 18:24 core
lrwxrwxrwx  1 root root        22 Aug  3 05:12 libtensorflow.so -> libtensorflow.so.2.2.0
lrwxrwxrwx  1 root root        22 Aug  3 05:12 libtensorflow.so.2 -> libtensorflow.so.2.2.0
-r-xr-xr-x  1 root root 182074912 Jul 31 18:05 libtensorflow.so.2.2.0
-r-xr-xr-x  1 root root    455903 Jul 31 15:05 libtensorflow.so.2.2.0-2.params
lrwxrwxrwx  1 root root        28 Jul 31 18:03 libtensorflow_framework.so -> libtensorflow_framework.so.2
lrwxrwxrwx  1 root root        32 Jul 31 18:03 libtensorflow_framework.so.2 -> libtensorflow_framework.so.2.2.0
-r-xr-xr-x  1 root root  37647000 Jul 31 18:03 libtensorflow_framework.so.2.2.0
-r-xr-xr-x  1 root root    197753 Jul 31 17:22 libtensorflow_framework.so.2.2.0-2.params
drwxr-xr-x  8 root root      4096 Jul 31 11:44 stream_executor
drwxr-xr-x  4 root root      4096 Jul 31 18:32 tools
/tmp/docker-share/tensorflow-java/tensorflow-core/tensorflow-core-api/build.sh: line 80: 31711 Illegal instruction     (core dumped) $BAZEL_BIN/java_op_generator --output_dir=$GEN_SRCS_DIR --api_dirs=$BAZEL_SRCS/external/org_tensorflow/tensorflow/core/api_def/base_api,src/bazel/api_def $TENSORFLOW_LIB
[INFO] Reactor Summary for TensorFlow Java Parent 0.2.0-SNAPSHOT:
[INFO] 
[INFO] TensorFlow Java Parent ............................. SUCCESS [  2.387 s]
[INFO] TensorFlow NdArray Library ......................... SUCCESS [03:04 min]
[INFO] TensorFlow Core Parent ............................. SUCCESS [  0.254 s]
[INFO] TensorFlow Core Annotation Processor ............... SUCCESS [  7.604 s]
[INFO] TensorFlow Core API Library ........................ FAILURE [05:32 min]
[INFO] TensorFlow Core API Library Platform ............... SKIPPED
[INFO] TensorFlow Framework Library ....................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  08:49 min

[karllessard]

Hi @aday00 ,

You are right, TF Java will build its own binaries of the TensorFlow runtime. Looking at your trace though, this build seems to be successful. What is failing is when we try to generate the operation wrappers for Java from our custom binary. That code is purely in C++ so it is unrelated with the version of JDK you are using.

Maybe disabling AVX left some operation definitions in a unexpected state, it is hard to say without seeing details of the core dump. Maybe you can rebuild only the generator in debug mode and see at which line it fails and for which op? i.e. invoking bazel for rebuilding :java_op_generator with -c dbg and reexecuting the binary like in the bash script in a debugger to output some trace?

[saudet]

Have you tried to remove all the AVX flags in the build.sh script and rebuild from scratch?

[aday00]

Thanks for your guidance @karllessard and @saudet! It seems to me we get the core dump from this build.sh line:

# Generate Java operator wrappers
$BAZEL_BIN/java_op_generator \
    --output_dir=$GEN_SRCS_DIR \
    --api_dirs=$BAZEL_SRCS/external/org_tensorflow/tensorflow/core/api_def/base_api,src/bazel/api_def \
    $TENSORFLOW_LIB

Likebuild.sh, the core dump is in tensorflow-core-api:

# ls
BUILD  WORKSPACE  bazel-bin  bazel-out  bazel-tensorflow-core-api  bazel-testlogs  build.sh  core  external  pom.xml  src
# file core
core: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from '/tmp/docker-share/tensorflow-java/tensorflow-core/tensorflow-core-api/bazel-bin', real uid: 0, effective uid: 0, real gid: 0, effective gid: 0, execfn: '/tmp/docker-share/tensorflow-java/tensorflow-core/tensorflow-core-api/bazel-bin/java_op_generator', platform: 'x86_64'
# pwd
/tmp/docker-share/tensorflow-java/tensorflow-core/tensorflow-core-api
# file bazel-bin
bazel-bin: symbolic link to /tmp/docker-share/.cache/bazel/_bazel_root/3221ddcd7458d3ec47b332d863b91a03/execroot/tensorflow_core_api/bazel-out/k8-opt/bin

Starting gdb on that core dump, we see the SIGILL and illegal instruction message....

# gdb bazel-bin/java_op_generator core
GNU gdb (Debian 8.2.1-2+b3) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bazel-bin/java_op_generator...(no debugging symbols found)...done.

warning: core file may not match specified executable file.
[New LWP 31973]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/tmp/docker-share/tensorflow-java/tensorflow-core/tensorflow-core-api/bazel-bin'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x00007f9960adca40 in nsync::nsync_mu_init(nsync::nsync_mu_s_*) ()
   from /tmp/docker-share/.cache/bazel/_bazel_root/3221ddcd7458d3ec47b332d863b91a03/execroot/tensorflow_core_api/bazel-out/k8-opt/bin/_solib_k8/_U_S_S_Cjava_Uop_Ugenerator___Uexternal_Sorg_Utensorflow_Stensorflow/libtensorflow_framework.so.2

A short backtrace from gdb...

(gdb) where
#0  0x00007f9960adca40 in nsync::nsync_mu_init(nsync::nsync_mu_s_*) ()
   from /tmp/docker-share/.cache/bazel/_bazel_root/3221ddcd7458d3ec47b332d863b91a03/execroot/tensorflow_core_api/bazel-out/k8-opt/bin/_solib_k8/_U_S_S_Cjava_Uop_Ugenerator___Uexternal_Sorg_Utensorflow_Stensorflow/libtensorflow_framework.so.2
#1  0x00007f995fdd3f5d in tensorflow::monitoring::Counter<2>* tensorflow::monitoring::Counter<2>::New<char const (&) [46], char const (&) [58], char const (&) [11], char const (&) [7]>(char const (&) [46], char const (&) [58], char const (&) [11], char const (&) [7]) ()
   from /tmp/docker-share/.cache/bazel/_bazel_root/3221ddcd7458d3ec47b332d863b91a03/execroot/tensorflow_core_api/bazel-out/k8-opt/bin/_solib_k8/_U_S_S_Cjava_Uop_Ugenerator___Uexternal_Sorg_Utensorflow_Stensorflow/libtensorflow_framework.so.2
#2  0x00007f995fdc11e1 in _GLOBAL__sub_I_loader.cc ()
   from /tmp/docker-share/.cache/bazel/_bazel_root/3221ddcd7458d3ec47b332d863b91a03/execroot/tensorflow_core_api/bazel-out/k8-opt/bin/_solib_k8/_U_S_S_Cjava_Uop_Ugenerator___Uexternal_Sorg_Utensorflow_Stensorflow/libtensorflow_framework.so.2
#3  0x00007f99613d637a in call_init (l=<optimized out>, argc=argc@entry=4, argv=argv@entry=0x7fffefa7cf38, env=env@entry=0x7fffefa7cf60) at dl-init.c:72
#4  0x00007f99613d6476 in call_init (env=0x7fffefa7cf60, argv=0x7fffefa7cf38, argc=4, l=<optimized out>) at dl-init.c:30
#5  _dl_init (main_map=0x7f99613f0190, argc=4, argv=0x7fffefa7cf38, env=0x7fffefa7cf60) at dl-init.c:119
#6  0x00007f99613c80ca in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#7  0x0000000000000004 in ?? ()
#8  0x00007fffefa7d4d9 in ?? ()
#9  0x00007fffefa7d53b in ?? ()
#10 0x00007fffefa7d555 in ?? ()
#11 0x00007fffefa7d60b in ?? ()
#12 0x0000000000000000 in ?? ()

And getting the full backtrace from gdb...

(gdb) bt full
#0  0x00007f9960adca40 in nsync::nsync_mu_init(nsync::nsync_mu_s_*) ()
   from /tmp/docker-share/.cache/bazel/_bazel_root/3221ddcd7458d3ec47b332d863b91a03/execroot/tensorflow_core_api/bazel-out/k8-opt/bin/_solib_k8/_U_S_S_Cjava_Uop_Ugenerator___Uexternal_Sorg_Utensorflow_Stensorflow/libtensorflow_framework.so.2
No symbol table info available.
#1  0x00007f995fdd3f5d in tensorflow::monitoring::Counter<2>* tensorflow::monitoring::Counter<2>::New<char const (&) [46], char const (&) [58], char const (&) [11], char const (&) [7]>(char const (&) [46], char const (&) [58], char const (&) [11], char const (&) [7]) ()
   from /tmp/docker-share/.cache/bazel/_bazel_root/3221ddcd7458d3ec47b332d863b91a03/execroot/tensorflow_core_api/bazel-out/k8-opt/bin/_solib_k8/_U_S_S_Cjava_Uop_Ugenerator___Uexternal_Sorg_Utensorflow_Stensorflow/libtensorflow_framework.so.2
No symbol table info available.
#2  0x00007f995fdc11e1 in _GLOBAL__sub_I_loader.cc ()
   from /tmp/docker-share/.cache/bazel/_bazel_root/3221ddcd7458d3ec47b332d863b91a03/execroot/tensorflow_core_api/bazel-out/k8-opt/bin/_solib_k8/_U_S_S_Cjava_Uop_Ugenerator___Uexternal_Sorg_Utensorflow_Stensorflow/libtensorflow_framework.so.2
No symbol table info available.
#3  0x00007f99613d637a in call_init (l=<optimized out>, argc=argc@entry=4, argv=argv@entry=0x7fffefa7cf38, env=env@entry=0x7fffefa7cf60) at dl-init.c:72
        j = <optimized out>
        jm = <optimized out>
        addrs = <optimized out>
        init_array = <optimized out>
#4  0x00007f99613d6476 in call_init (env=0x7fffefa7cf60, argv=0x7fffefa7cf38, argc=4, l=<optimized out>) at dl-init.c:30
        init_array = <optimized out>
        init_array = <optimized out>
        j = <optimized out>
        jm = <optimized out>
        addrs = <optimized out>
#5  _dl_init (main_map=0x7f99613f0190, argc=4, argv=0x7fffefa7cf38, env=0x7fffefa7cf60) at dl-init.c:119
        preinit_array = <optimized out>
        preinit_array_size = <optimized out>
        i = 1
#6  0x00007f99613c80ca in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
        library_path = 0x0
        version_info = 0
        any_debug = 0
        _dl_rtld_libname = {name = 0x55e50d2d2238 "/lib64/ld-linux-x86-64.so.2", next = 0x7f99613f0000 <newname>, dont_free = 0}
        relocate_time = 112990700
        _dl_rtld_libname2 = {name = 0x0, next = 0x0, dont_free = 0}
        start_time = 11281447588560320
        tls_init_tp_called = true
        load_time = 47901180
        audit_list = 0x0
        preloadlist = 0x0
        __GI__dl_argv = 0x7fffefa7cf38
        _dl_argc = 4
        audit_list_string = 0x0
        _rtld_global = {_dl_ns = {{_ns_loaded = 0x7f99613f0190, _ns_nloaded = 11, _ns_main_searchlist = 0x7f99613f0450, _ns_global_scope_alloc = 0, _ns_unique_sym_table = {lock = {
                  mutex = pthread_mutex_t = {Type = Recursive, Status = Not acquired, Robust = No, Shared = No, Protocol = None}}, entries = 0x7f995f327000, size = 509, n_elements = 235, 
                free = 0x7f99613dff00 <free>}, _ns_debug = {r_version = 0, r_map = 0x0, r_brk = 0, r_state = RT_CONSISTENT, r_ldbase = 0}}, {_ns_loaded = 0x0, _ns_nloaded = 0, 
              _ns_main_searchlist = 0x0, _ns_global_scope_alloc = 0, _ns_unique_sym_table = {lock = {mutex = pthread_mutex_t = {Type = Normal, Status = Not acquired, Robust = No, 
                    Shared = No, Protocol = None}}, entries = 0x0, size = 0, n_elements = 0, free = 0x0}, _ns_debug = {r_version = 0, r_map = 0x0, r_brk = 0, r_state = RT_CONSISTENT, 
                r_ldbase = 0}} <repeats 15 times>}, _dl_nns = 1, _dl_load_lock = {mutex = pthread_mutex_t = {Type = Recursive, Status = Not acquired, Robust = No, Shared = No, 
              Protocol = None}}, _dl_load_write_lock = {mutex = pthread_mutex_t = {Type = Recursive, Status = Not acquired, Robust = No, Shared = No, Protocol = None}}, _dl_load_adds = 11, 
          _dl_initfirst = 0x0, _dl_cpuclock_offset = 11281447588593660, _dl_profile_map = 0x0, _dl_num_relocations = 50730, _dl_num_cache_relocations = 15992, 
          _dl_all_dirs = 0x7f99613c54d0, _dl_rtld_map = {l_addr = 140296738074624, l_name = 0x55e50d2d2238 "/lib64/ld-linux-x86-64.so.2", l_ld = 0x7f99613eee78, l_next = 0x7f995f53afd0, 
            l_prev = 0x7f995f53aa10, l_real = 0x7f99613ef9f0 <_rtld_global+2448>, l_ns = 0, l_libname = 0x7f99613f0050 <_dl_rtld_libname>, l_info = {0x0, 0x0, 0x7f99613eeef8, 
              0x7f99613eeee8, 0x7f99613eee88, 0x7f99613eeea8, 0x7f99613eeeb8, 0x7f99613eef28, 0x7f99613eef38, 0x7f99613eef48, 0x7f99613eeec8, 0x7f99613eeed8, 0x0, 0x0, 0x7f99613eee78, 0x0, 
              0x0, 0x0, 0x0, 0x0, 0x7f99613eef08, 0x0, 0x0, 0x7f99613eef18, 0x0 <repeats 13 times>, 0x7f99613eef68, 0x7f99613eef58, 0x0, 0x0, 0x7f99613eef88, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 
              0x0, 0x0, 0x7f99613eef78, 0x0 <repeats 25 times>, 0x7f99613eee98}, l_phdr = 0x7f99613c7040, l_entry = 0, l_phnum = 9, l_ldnum = 0, l_searchlist = {r_list = 0x0, r_nlist = 0}, 
            l_symbolic_searchlist = {r_list = 0x0, r_nlist = 0}, l_loader = 0x0, l_versions = 0x7f995f332b50, l_nversions = 6, l_nbuckets = 17, l_gnu_bitmask_idxbits = 3, l_gnu_shift = 8, 
            l_gnu_bitmask = 0x7f99613c7348, {l_gnu_buckets = 0x7f99613c7368, l_chain = 0x7f99613c7368}, {l_gnu_chain_zero = 0x7f99613c73a8, l_buckets = 0x7f99613c73a8}, 
            l_direct_opencount = 0, l_type = lt_library, l_relocated = 1, l_init_called = 1, l_global = 1, l_reserved = 0, l_phdr_allocated = 0, l_soname_added = 0, l_faked = 0, 
            l_need_tls_init = 0, l_auditing = 0, l_audit_any_plt = 0, l_removed = 0, l_contiguous = 0, l_symbolic_in_local_scope = 0, l_free_initfini = 0, l_cet = lc_unknown, 
            l_rpath_dirs = {dirs = 0x0, malloced = 0}, l_reloc_result = 0x0, l_versyms = 0x7f99613c7984, l_origin = 0x0, l_map_start = 140296738074624, l_map_end = 140296738242960, 
            l_text_end = 140296738200352, l_scope_mem = {0x0, 0x0, 0x0, 0x0}, l_scope_max = 0, l_scope = 0x0, l_local_scope = {0x0, 0x0}, l_file_id = {dev = 0, ino = 0}, l_runpath_dirs = {
              dirs = 0x0, malloced = 0}, l_initfini = 0x0, l_reldeps = 0x0, l_reldepsmax = 0, l_used = 1, l_feature_1 = 0, l_flags_1 = 0, l_flags = 0, l_idx = 0, l_mach = {plt = 0, 
              gotplt = 0, tlsdesc_table = 0x0}, l_lookup_cache = {sym = 0x7f99613c74f0, type_class = 1, value = 0x7f995f53aa10, ret = 0x7f995f3431f0}, l_tls_initimage = 0x0, 
            l_tls_initimage_size = 0, l_tls_blocksize = 0, l_tls_align = 0, l_tls_firstbyte_offset = 0, l_tls_offset = 0, l_tls_modid = 0, l_tls_dtor_count = 0, l_relro_addr = 161344, 
            l_relro_size = 2496, l_serial = 0, l_audit = 0x7f99613efe68 <_rtld_global+3592>}, audit_data = {{cookie = 0, bindflags = 0} <repeats 16 times>}, 
          _dl_rtld_lock_recursive = 0x7f995f5236c0 <__GI___pthread_mutex_lock>, _dl_rtld_unlock_recursive = 0x7f995f524fa0 <__GI___pthread_mutex_unlock>, _dl_x86_feature_1 = {0, 0}, 
          _dl_x86_legacy_bitmap = {0, 0}, _dl_make_stack_executable_hook = 0x7f995f520380 <__make_stacks_executable>, _dl_stack_flags = 6, _dl_tls_dtv_gaps = false, 
          _dl_tls_max_dtv_idx = 3, _dl_tls_dtv_slotinfo_list = 0x7f995f332d50, _dl_tls_static_nelem = 3, _dl_tls_static_size = 4608, _dl_tls_static_used = 616, _dl_tls_static_align = 64, 
          _dl_initial_dtv = 0x7f995f330260, _dl_tls_generation = 1, _dl_init_static_tls = 0x7f995f520a60 <__pthread_init_static_tls>, 
          _dl_wait_lookup_done = 0x7f995f520b80 <__wait_lookup_done>, _dl_scope_free_list = 0x0}
        _rtld_global_ro = {_dl_debug_mask = 0, _dl_osversion = 264558, _dl_platform = 0x7fffefa7d239 "x86_64", _dl_platformlen = 6, _dl_pagesize = 4096, _dl_inhibit_cache = 0, 
          _dl_initial_searchlist = {r_list = 0x7f995f53b6c8, r_nlist = 10}, _dl_clktck = 100, _dl_verbose = 0, _dl_debug_fd = 2, _dl_lazy = 1, _dl_bind_not = 0, _dl_dynamic_weak = 0, 
          _dl_fpu_control = 895, _dl_correct_cache_id = 771, _dl_hwcap = 2, _dl_auxv = 0x7fffefa7d0f0, _dl_x86_cpu_features = {kind = arch_kind_intel, max_cpuid = 11, cpuid = {{
                eax = 263875, ebx = 68159488, ecx = 1138287551, edx = 3219913727}, {eax = 0, ebx = 8834, ecx = 0, edx = 0}, {eax = 0, ebx = 0, ecx = 257, edx = 672139264}}, family = 6, 
            model = 76, xsave_state_size = 0, xsave_state_full_size = 0, feature = {1360432}, data_cache_size = 0, shared_cache_size = 0, non_temporal_threshold = 0}, 
          _dl_x86_hwcap_flags = {"sse2\000\000\000\000", "x86_64\000\000", "avx512_1"}, _dl_x86_platforms = {"i586\000\000\000\000", "i686\000\000\000\000", "haswell\000", "xeon_phi"}, 
          _dl_inhibit_rpath = 0x0, _dl_origin_path = 0x0, _dl_use_load_bias = 0, _dl_profile = 0x0, _dl_profile_output = 0x7f99613e607c "/var/tmp", _dl_trace_prelink = 0x0, 
          _dl_trace_prelink_map = 0x0, _dl_init_all_dirs = 0x7f99613c54d0, _dl_sysinfo_dso = 0x7fffefbcb000, _dl_sysinfo_map = 0x7f99613f0730, _dl_hwcap2 = 0, 
          _dl_debug_printf = 0x7f99613d7090 <_dl_debug_printf>, _dl_mcount = 0x7f99613d8460 <__GI__dl_mcount>, _dl_lookup_symbol_x = 0x7f99613d1240 <_dl_lookup_symbol_x>, 
          _dl_open = 0x7f99613d9b20 <_dl_open>, _dl_close = 0x7f99613db840 <_dl_close>, _dl_tls_get_addr_soft = 0x7f99613d9390 <_dl_tls_get_addr_soft>, 
          _dl_discover_osversion = 0x7f99613dfaf0 <_dl_discover_osversion>, _dl_audit = 0x0, _dl_naudit = 0}
        _dl_skip_args = 0
        __pointer_chk_guard_local = 12940881931090418409
#7  0x0000000000000004 in ?? ()
No symbol table info available.
#8  0x00007fffefa7d4d9 in ?? ()
No symbol table info available.
#9  0x00007fffefa7d53b in ?? ()
No symbol table info available.
#10 0x00007fffefa7d555 in ?? ()
No symbol table info available.
#11 0x00007fffefa7d60b in ?? ()
No symbol table info available.
#12 0x0000000000000000 in ?? ()
No symbol table info available.
```

[aday00]

To remove AVX from build.sh, kindly advise if I should make this change? My CPU also lacks FMA, so I should probably remove that too? This is not my expertise.

--- build.sh    2020-07-29 18:34:41.752466899 -0400
+++ build.no-avx.sh     2020-08-05 14:38:09.157969431 -0400
@@ -13,7 +13,7 @@
     export PATH=$PATH:$(pwd)/bazel-tensorflow-core-api/external/mkl_windows/lib/
     export PYTHON_BIN_PATH=$(which python.exe)
 else
-    export BUILD_FLAGS="--copt=-msse4.1 --copt=-msse4.2 --copt=-mavx `#--copt=-mavx2 --copt=-mfma` --cxxopt=-std=c++14 --host_cxxopt=-std=c++14 --linkopt=-lstdc++ --host_linkopt=-lstdc++"
+    export BUILD_FLAGS="--copt=-msse4.1 --copt=-msse4.2 --cxxopt=-std=c++14 --host_cxxopt=-std=c++14 --linkopt=-lstdc++ --host_linkopt=-lstdc++"
     export PYTHON_BIN_PATH=$(which python3)
 fi

[karllessard]

Thanks @aday00 , so it looks like the illegal instruction is happening in the core library, not the generator code itself, which is good news. So it is probably an incompatibility with machine and the compilation options when building the library, the changes you propose in the build script seem accurate.

[aday00]

Thanks @karllessard and @saudet! That builds.

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for TensorFlow Java Parent 0.2.0-SNAPSHOT:
[INFO]
[INFO] TensorFlow Java Parent ............................. SUCCESS [  2.612 s]
[INFO] TensorFlow NdArray Library ......................... SUCCESS [03:04 min]
[INFO] TensorFlow Core Parent ............................. SUCCESS [  0.155 s]
[INFO] TensorFlow Core Annotation Processor ............... SUCCESS [  7.553 s]
[INFO] TensorFlow Core API Library ........................ SUCCESS [  13:30 h]
[INFO] TensorFlow Core API Library Platform ............... SUCCESS [  3.025 s]
[INFO] TensorFlow Framework Library ....................... SUCCESS [ 45.052 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  13:34 h
[INFO] Finished at: 2020-08-06T08:41:18Z
[INFO] ------------------------------------------------------------------------

I'm fine closing this, but do you want a patch to compile with the right flags, depending on /proc/cpuinfo? The patched build.sh could compile with AVX512 if available, and not compile with AVX if unavailable. I won't propose an AVX512-related patch unless I can test it, and I may have access to a CPU w/ AVX512 in a few weeks, but not sure yet.

$ diff -u build.sh build.no-avx.sh 
--- build.sh    2020-08-05 15:02:11.934001156 -0400
+++ build.no-avx.sh     2020-08-06 05:01:43.766500118 -0400
@@ -13,7 +13,11 @@
     export PATH=$PATH:$(pwd)/bazel-tensorflow-core-api/external/mkl_windows/lib/
     export PYTHON_BIN_PATH=$(which python.exe)
 else
-    export BUILD_FLAGS="--copt=-msse4.1 --copt=-msse4.2 --cxxopt=-std=c++14 --host_cxxopt=-std=c++14 --linkopt=-lstdc++ --host_linkopt=-lstdc++"
+    if [[ -n "$(grep -l avx2 /proc/cpuinfo)" ]]; then
+      export BUILD_FLAGS="--copt=-msse4.1 --copt=-msse4.2 --copt=-mavx `#--copt=-mavx2 --copt=-mfma` --cxxopt=-std=c++14 --host_cxxopt=-std=c++14 --linkopt=-lstdc++ --host_linkopt=-lstdc++"
+    else
+      export BUILD_FLAGS="--copt=-msse4.1 --copt=-msse4.2 --cxxopt=-std=c++14 --host_cxxopt=-std=c++14 --linkopt=-lstdc++ --host_linkopt=-lstdc++"
+    fi
     export PYTHON_BIN_PATH=$(which python3)
 fi

Thanks again for this great project, and all your help!

[karllessard]

Wonderful, good luck!

[saudet]

I think it makes sense to autodetect AVX, but maybe we can disable it from the plain build and let MKL-DNN/DNNL do the rest? That would make the binaries useful to all those users using these cheap embedded chips. @karllessard

[karllessard]

Good idea @saudet , let’s do this, at least we have a build that is safe on all CPUs.

[aday00]

For what it's worth, Tensorflow by default builds w/ AVX (and is therefore broken for my cheap CPU w/o AVX), so Tensorflow-Java is no worse than Tensorflow itself: tensorflow/tensorflow#18689

The AVX build decision is probably more important for Tensorflow, because Google distributes a CPU-only .whl package that is intended to work for most environments. I have a cheap CPU as a test environment, before scaling up.

Soon I'll run on multi-CPU-socket & quad-GPU dev systems, and it would be nice if mvn install detected if AVX2 or AVX512 were available and did the right thing, since we're going through the trouble of building from source rather than relying on a pre-built .whl package. It's good to be fast and use AVX2 or AVX512 if available, because I'll probably make CPU-only kernels in Tensorflow-Java for preprocessing data, before GPU-based Tensorflow handles deep learning.

I'm not familiar with Intel MKL/BLAS/LAPACK. I'm not sure if users should be expected to pull in that dependency to use AVX. I don't know how much slower Tensorflow-Java becomes by disabling AVX, but I wouldn't want to slow things down for almost everyone else just because I'm testing on a cheap CPU at the moment. Thanks for letting me share my uninformed perspective!

[aday00]

I made a fork, with a branch to auto-detect AVX for this ticket and @karllessard's session-based model save code (#100) https://github.com/aday00/java/tree/session_saved_model.

Currently this branch just auto-detects AVX aday00@d9a4105.

Will test session-based model save soon, with some minor code cleanup. Thanks again!