Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linux: compact_dex_converter fails with "Illegal instruction (core dumped)" #29

Closed
superr opened this issue Sep 7, 2018 · 18 comments
Closed

Comments

@superr
Copy link

superr commented Sep 7, 2018

Hello,

When trying to convert cdex to dex, compact_dex_converter fails:

$ bin/compact_dex_converterd services_classes.cdex 
Opened 'services_classes.cdex', DEX version '001'
Illegal instruction (core dumped)

You asked if I would use valgrind to run it, and to my surprise it works when run through valgrind:

$ valgrind bin/compact_dex_converterd services_classes.cdex 
==6242== Memcheck, a memory error detector
==6242== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==6242== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==6242== Command: bin/compact_dex_converterd services_classes.cdex
==6242== 
--6242-- Warning: DWARF2 CFI reader: unhandled DW_OP_ opcode 0x13
--6242-- Warning: DWARF2 CFI reader: unhandled DW_OP_ opcode 0x13
Opened 'services_classes.cdex', DEX version '001'
compact_dex_converterd I 09-07 08:17:15  6242  6242 compact_dex_converter_main.cc:172] StandardDex file successfully extracted to services_classes.cdex.new
==6242== 
==6242== HEAP SUMMARY:
==6242==     in use at exit: 264 bytes in 7 blocks
==6242==   total heap usage: 1,377,216 allocs, 1,377,209 frees, 115,375,932 bytes allocated
==6242== 
==6242== LEAK SUMMARY:
==6242==    definitely lost: 0 bytes in 0 blocks
==6242==    indirectly lost: 0 bytes in 0 blocks
==6242==      possibly lost: 0 bytes in 0 blocks
==6242==    still reachable: 264 bytes in 7 blocks
==6242==         suppressed: 0 bytes in 0 blocks
==6242== Rerun with --leak-check=full to see details of leaked memory
==6242== 
==6242== For counts of detected and suppressed errors, rerun with: -v
==6242== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Now I am really confused

@anestisb
Copy link
Owner

anestisb commented Sep 7, 2018

Xmm weird. Can you try attaching gdb and see if you can reproduce the crash? If you're not familiar with gdb use the following steps:

  1. Install gdb (sudo apt-get install gdb)
  2. Run binary from gdb ( gdb bin/compact_dex_converterd). Notice that no arguments are provided yet.
  3. When gdb finished initialising (prompt open for next command) run the against the input file (run services_classes.cdex)
  4. When the execution stops with the signal error grab the backtrace by typing bt in the prompt
  5. Copy the entire output (from the moment you launched gdb) here

@superr
Copy link
Author

superr commented Sep 7, 2018

Thank you for your response. Here you go:

$ gdb bin/compact_dex_converterd 
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bin/compact_dex_converterd...done.
(gdb) run services_classes.cdex
Starting program: /home/superr/android/kitchen/SRKP/tmp/bin/compact_dex_converterd services_classes.cdex
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Opened 'services_classes.cdex', DEX version '001'

Program received signal SIGILL, Illegal instruction.
0x00007ffff7f9fd37 in std::__1::allocator<unsigned char>::construct<unsigned char, unsigned char const&> (__args=<optimized out>, this=<optimized out>, 
    __p=<optimized out>) at external/libcxx/include/memory:1805
1805	external/libcxx/include/memory: No such file or directory.
(gdb) bt
#0  0x00007ffff7f9fd37 in std::__1::allocator<unsigned char>::construct<unsigned char, unsigned char const&> (__args=<optimized out>, this=<optimized out>, 
    __p=<optimized out>) at external/libcxx/include/memory:1805
#1  std::__1::allocator_traits<std::__1::allocator<unsigned char> >::__construct<unsigned char, unsigned char const&> (__a=..., __p=<optimized out>, 
    __args=<optimized out>) at external/libcxx/include/memory:1715
#2  std::__1::allocator_traits<std::__1::allocator<unsigned char> >::construct<unsigned char, unsigned char const&> (__p=<optimized out>, __a=..., 
    __args=<optimized out>) at external/libcxx/include/memory:1561
#3  std::__1::__split_buffer<unsigned char, std::__1::allocator<unsigned char>&>::__construct_at_end (__n=<optimized out>, __x=<optimized out>, 
    this=<optimized out>) at external/libcxx/include/__split_buffer:222
#4  std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >::__append (this=0x5555595f5c00, __n=47, __x=@0x7fffffffd6e7: 0 '\000')
    at external/libcxx/include/vector:1067
#5  0x00007ffff7f9fa89 in std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >::resize (this=0x0, __sz=<optimized out>, __x=<optimized out>)
    at external/libcxx/include/vector:2019
#6  art::DexContainer::VectorSection::Resize (this=<optimized out>, 
    size=<optimized out>) at art/dexlayout/dex_container.h:69
#7  0x00007ffff7fd71aa in art::DexWriter::Stream::EnsureStorage (length=0, 
    this=<optimized out>) at art/dexlayout/dex_writer.h:159
#8  art::DexWriter::Stream::Seek (position=93825060003008, 
---Type <return> to continue, or q <return> to quit---
    this=<optimized out>) at art/dexlayout/dex_writer.h:87
#9  art::DexWriter::Write (this=0x5555595f5580, output=0x5555595f5bf0, 
    error_msg=<optimized out>) at art/dexlayout/dex_writer.cc:798
#10 0x00007ffff7fd77bd in art::DexWriter::Output (dex_layout=<optimized out>, 
    container=0x7fffffffdaa8, compute_offsets=true, error_msg=0x7fffffffdb40)
    at art/dexlayout/dex_writer.cc:929
#11 0x00007ffff7fa9e75 in art::DexLayout::OutputDexFile (this=0x7fffffffd820, 
    input_dex_file=0x7fffffffd8d0, compute_offsets=<optimized out>, 
    dex_container=0x7fffffffe173, error_msg=0x7fffffffd8e0)
    at art/dexlayout/dexlayout.cc:1841
#12 0x00007ffff7faa33a in art::DexLayout::ProcessDexFile (
    this=<optimized out>, file_name=<optimized out>, dex_file=0x55555555d7e0, 
    dex_file_index=<optimized out>, dex_container=0x7fffffffdaa8, 
    error_msg=0x7fffffffdb40) at art/dexlayout/dexlayout.cc:1920
#13 0x0000555555556dcc in art::ProcessFile (
    file_name=0x7fffffffe173 "services_classes.cdex", 
    out_file=<optimized out>, output_dex_directory=<optimized out>, 
    verify_output_file=<optimized out>)
    at art/dexlayout/compact_dex_converter_main.cc:111
#14 art::DexlayoutDriver (argc=2, argv=<optimized out>)
    at art/dexlayout/compact_dex_converter_main.cc:235
#15 0x0000555555557dbd in main (argc=2, argv=0x7fffffffdd98)
    at art/dexlayout/compact_dex_converter_main.cc:252

EDIT: Got the full output now. Sorry about that

@anestisb
Copy link
Owner

anestisb commented Sep 8, 2018

@superr it seems that the libc++.so that I have compiled and shared is not compatible with your system due to some CPU missing features (most probably an old model not supporting all new instruction emitted from newer compiler version).

The reason that it works when running from within valgrind, is because valgrind places its own hooks to instrument allocations (including the one that contains the illegal instruction) and thus the original implementation is never invoked (SIGILL is not triggered).

Unfortunately I cannot do much from my side. However, you can try forking the necessary AOSP repos, apply the compact_dex_converter patch in the ART git repo and build yourself. Hopefully, when compiling libc++ the compiler will handle the older CPU issue.

@superr
Copy link
Author

superr commented Sep 9, 2018

Thank you for your help :)

I have synced the Pie repo and applied the patch. I am not sure how to get the compact_dex_converter binary compiled from here.

I prefer a static binary, but anything that will work on my PC would be great! Thanks :)

@superr
Copy link
Author

superr commented Sep 10, 2018

I think I have figured out how to compile compact_dex_converter. However, it returns this error:

FAILED: out/soong/.intermediates/art/dexlayout/compact_dex_converter/android_arm_armv7-a-neon_core/obj/art/dexlayout/compact_dex_converter_main.o
PWD=/proc/self/cwd prebuilts/clang/host/linux-x86/clang-r328903/bin/clang++ -c -Ibionic/libc/private -Iexternal/vixl/src -Iart/dexlayout -mthumb -Os -fomit-frame-pointer -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -no-canonical-prefixes -DNDEBUG -UDEBUG -fno-exceptions -Wno-multichar -O2 -g -fno-strict-aliasing -fdebug-prefix-map=/proc/self/cwd= -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -Wno-expansion-to-defined -Wno-zero-as-null-pointer-constant -Wno-deprecated-register -Wno-sign-compare -ffunction-sections -fdata-sections -fno-short-enums -funwind-tables -fstack-protector-strong -Wa,--noexecstack -D_FORTIFY_SOURCE=2 -Wstrict-aliasing=2 -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -Werror=format-security -nostdlibinc -msoft-float -march=armv7-a -mfloat-abi=softfp -mfpu=neon  -Isystem/core/base/include -Iart/runtime -Iart/libdexfile -Iart/libartbase -Iout/soong/.intermediates/art/tools/cpp-define-generator/cpp-define-generator-asm-support/gen -Iart/dexlayout -Iexternal/libcxx/include -Iexternal/libcxxabi/include -Ibionic/libc/system_properties/include -Isystem/core/property_service/libpropertyinfoparser/include -Isystem/core/include -Isystem/media/audio/include -Ihardware/libhardware/include -Ihardware/libhardware_legacy/include -Ihardware/ril/include -Ilibnativehelper/include -Iframeworks/native/include -Iframeworks/native/opengl/include -Iframeworks/av/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libc/kernel/android/scsi -isystem bionic/libc/kernel/android/uapi -Ilibnativehelper/include_jni -fno-rtti -ggdb3 -Wall -Werror -Wextra -Wstrict-aliasing -fstrict-aliasing -Wunreachable-code -Wredundant-decls -Wshadow -Wunused -fvisibility=protected -Wthread-safety -Wthread-safety-negative -Wimplicit-fallthrough -Wfloat-equal -Wint-to-void-pointer-cast -Wused-but-marked-unused -Wdeprecated -Wunreachable-code-break -Wunreachable-code-return -D_LIBCPP_ENABLE_THREAD_SAFETY_ANNOTATIONS -O3 -DART_DEFAULT_GC_TYPE_IS_CMS -DIMT_SIZE=43 -DART_USE_READ_BARRIER=1 -DART_READ_BARRIER_TYPE_IS_BAKER=1 -DART_DEFAULT_COMPACT_DEX_LEVEL=fast -DART_STACK_OVERFLOW_GAP_arm=8192 -DART_STACK_OVERFLOW_GAP_arm64=8192 -DART_STACK_OVERFLOW_GAP_mips=16384 -DART_STACK_OVERFLOW_GAP_mips64=16384 -DART_STACK_OVERFLOW_GAP_x86=8192 -DART_STACK_OVERFLOW_GAP_x86_64=8192 -DUSE_D8_DESUGAR=1 -Wmissing-noreturn -DART_TARGET -DART_ENABLE_CODEGEN_arm -Wframe-larger-than=1736 -DART_FRAME_SIZE_LIMIT=1736 -DART_BASE_ADDRESS=0x70000000 -DART_TARGET_ANDROID -DART_BASE_ADDRESS_MIN_DELTA=-0x1000000 -DART_BASE_ADDRESS_MAX_DELTA=0x1000000 -target armv7a-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin -DANDROID_STRICT -fPIE -D_USING_LIBCXX -std=gnu++14 -Wsign-promo -Wno-inconsistent-missing-override -Wno-null-dereference -D_LIBCPP_ENABLE_THREAD_SAFETY_ANNOTATIONS -Wno-thread-safety-negative -Wno-gnu-include-next -fvisibility-inlines-hidden -fno-rtti  -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=address-of-temporary -Werror=return-type -Wno-tautological-constant-compare -Wno-tautological-type-limit-compare -Wno-tautological-unsigned-enum-zero-compare -Wno-tautological-unsigned-zero-compare -Wno-null-pointer-arithmetic -Wno-c++98-compat-extra-semi -MD -MF out/soong/.intermediates/art/dexlayout/compact_dex_converter/android_arm_armv7-a-neon_core/obj/art/dexlayout/compact_dex_converter_main.o.d -o out/soong/.intermediates/art/dexlayout/compact_dex_converter/android_arm_armv7-a-neon_core/obj/art/dexlayout/compact_dex_converter_main.o art/dexlayout/compact_dex_converter_main.cc
art/dexlayout/compact_dex_converter_main.cc:39:10: fatal error: 'mem_map.h' file not found
#include "mem_map.h"
         ^~~~~~~~~~~
1 error generated.
14:31:01 ninja failed with: exit status 1
build/make/core/main.mk:21: recipe for target 'run_soong_ui' failed
make: *** [run_soong_ui] Error 1

Any idea how to fix this?

I synced the full master branch before patching and compiling with the command make compact_dex_converter from the source tree root.

EDIT: After searching for mem_map.h in the source tree:

$ find . -name "mem_map.h" 2>/dev/null
./art/libartbase/base/mem_map.h

Thank you for your assistance :)

@superr
Copy link
Author

superr commented Sep 10, 2018

I was able to get past the first error by changing in file compact_dex_converter_main.cc:

This:

#include "mem_map.h"

To:

#include "base/mem_map.h"

Unfortunately there is a new error:

FAILED: out/soong/.intermediates/art/dexlayout/compact_dex_converter/android_arm_armv7-a-neon_core/obj/art/dexlayout/compact_dex_converter_main.o
PWD=/proc/self/cwd prebuilts/clang/host/linux-x86/clang-r328903/bin/clang++ -c -Ibionic/libc/private -Iexternal/vixl/src -Iart/dexlayout -mthumb -Os -fomit-frame-pointer -DANDROID -fmessage-length=0 -W -Wall -Wno-unused -Winit-self -Wpointer-arith -no-canonical-prefixes -DNDEBUG -UDEBUG -fno-exceptions -Wno-multichar -O2 -g -fno-strict-aliasing -fdebug-prefix-map=/proc/self/cwd= -D__compiler_offsetof=__builtin_offsetof -Werror=int-conversion -Wno-reserved-id-macro -Wno-format-pedantic -Wno-unused-command-line-argument -fcolor-diagnostics -Wno-expansion-to-defined -Wno-zero-as-null-pointer-constant -Wno-deprecated-register -Wno-sign-compare -ffunction-sections -fdata-sections -fno-short-enums -funwind-tables -fstack-protector-strong -Wa,--noexecstack -D_FORTIFY_SOURCE=2 -Wstrict-aliasing=2 -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Werror=date-time -Werror=format-security -nostdlibinc -msoft-float -march=armv7-a -mfloat-abi=softfp -mfpu=neon  -Isystem/core/base/include -Iart/runtime -Iart/libdexfile -Iart/libartbase -Iout/soong/.intermediates/art/tools/cpp-define-generator/cpp-define-generator-asm-support/gen -Iart/dexlayout -Iexternal/libcxx/include -Iexternal/libcxxabi/include -Ibionic/libc/system_properties/include -Isystem/core/property_service/libpropertyinfoparser/include -Isystem/core/include -Isystem/media/audio/include -Ihardware/libhardware/include -Ihardware/libhardware_legacy/include -Ihardware/ril/include -Ilibnativehelper/include -Iframeworks/native/include -Iframeworks/native/opengl/include -Iframeworks/av/include -isystem bionic/libc/include -isystem bionic/libc/kernel/uapi -isystem bionic/libc/kernel/uapi/asm-arm -isystem bionic/libc/kernel/android/scsi -isystem bionic/libc/kernel/android/uapi -Ilibnativehelper/include_jni -fno-rtti -ggdb3 -Wall -Werror -Wextra -Wstrict-aliasing -fstrict-aliasing -Wunreachable-code -Wredundant-decls -Wshadow -Wunused -fvisibility=protected -Wthread-safety -Wthread-safety-negative -Wimplicit-fallthrough -Wfloat-equal -Wint-to-void-pointer-cast -Wused-but-marked-unused -Wdeprecated -Wunreachable-code-break -Wunreachable-code-return -D_LIBCPP_ENABLE_THREAD_SAFETY_ANNOTATIONS -O3 -DART_DEFAULT_GC_TYPE_IS_CMS -DIMT_SIZE=43 -DART_USE_READ_BARRIER=1 -DART_READ_BARRIER_TYPE_IS_BAKER=1 -DART_DEFAULT_COMPACT_DEX_LEVEL=fast -DART_STACK_OVERFLOW_GAP_arm=8192 -DART_STACK_OVERFLOW_GAP_arm64=8192 -DART_STACK_OVERFLOW_GAP_mips=16384 -DART_STACK_OVERFLOW_GAP_mips64=16384 -DART_STACK_OVERFLOW_GAP_x86=8192 -DART_STACK_OVERFLOW_GAP_x86_64=8192 -DUSE_D8_DESUGAR=1 -Wmissing-noreturn -DART_TARGET -DART_ENABLE_CODEGEN_arm -Wframe-larger-than=1736 -DART_FRAME_SIZE_LIMIT=1736 -DART_BASE_ADDRESS=0x70000000 -DART_TARGET_ANDROID -DART_BASE_ADDRESS_MIN_DELTA=-0x1000000 -DART_BASE_ADDRESS_MAX_DELTA=0x1000000 -target armv7a-linux-androideabi -Bprebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/arm-linux-androideabi/bin -DANDROID_STRICT -fPIE -D_USING_LIBCXX -std=gnu++14 -Wsign-promo -Wno-inconsistent-missing-override -Wno-null-dereference -D_LIBCPP_ENABLE_THREAD_SAFETY_ANNOTATIONS -Wno-thread-safety-negative -Wno-gnu-include-next -fvisibility-inlines-hidden -fno-rtti  -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=address-of-temporary -Werror=return-type -Wno-tautological-constant-compare -Wno-tautological-type-limit-compare -Wno-tautological-unsigned-enum-zero-compare -Wno-tautological-unsigned-zero-compare -Wno-null-pointer-arithmetic -Wno-c++98-compat-extra-semi -MD -MF out/soong/.intermediates/art/dexlayout/compact_dex_converter/android_arm_armv7-a-neon_core/obj/art/dexlayout/compact_dex_converter_main.o.d -o out/soong/.intermediates/art/dexlayout/compact_dex_converter/android_arm_armv7-a-neon_core/obj/art/dexlayout/compact_dex_converter_main.o art/dexlayout/compact_dex_converter_main.cc
art/dexlayout/compact_dex_converter_main.cc:81:42: error: too few arguments to function call, expected 8, have 7
                               &dex_files)) {
                                         ^
art/libdexfile/dex/dex_file_loader.h:149:3: note: 'OpenAll' declared here
  virtual bool OpenAll(const uint8_t* base,
  ^
1 error generated.
15:16:26 ninja failed with: exit status 1
build/make/core/main.mk:21: recipe for target 'run_soong_ui' failed
make: *** [run_soong_ui] Error 1

Sorry for all the posts, just trying to make my way through this. Thanks again for your help :)

@Furniel
Copy link

Furniel commented Sep 11, 2018

@superr

--- a/compact_dex_converter_main.cc	2018-09-11 09:14:21.928715800 +0300
+++ b/compact_dex_converter_main.cc	2018-09-03 23:56:40.856742000 +0300
@@ -36,7 +36,7 @@
 #include "base/os.h"
 #include "base/unix_file/fd_file.h"
 #include "dex/dex_file_loader.h"
-#include "mem_map.h"
+#include "base/mem_map.h"
 #include "runtime.h"
 
 namespace art {
@@ -67,19 +67,22 @@
 
   // We cannot use the ArtDexFileLoader since it only supports openning CompactDex
   // files from Vdex containers.
-  std::string error;
+  DexFileLoaderErrorCode error_code;
+  std::string error_msg;
   const bool kVerifyChecksum = false;
   const bool kVerify = true;  // Input file should always verify
   const DexFileLoader dex_file_loader;
   std::vector<std::unique_ptr<const DexFile>> dex_files;
   if (!dex_file_loader.OpenAll(reinterpret_cast<const uint8_t*>(content.data()),
                                content.size(),
                                file_name,
                                kVerify,
                                kVerifyChecksum,
-                               &error,
+                               &error_code,
+                               &error_msg,
                                &dex_files)) {
-    LOG(ERROR) << "Input Dex file open failed: " << error;
+    LOG(ERROR) << "Input Dex file open failed: " << error_msg;
     return 1;
   }
 
@@ -112,8 +115,8 @@
                                  dex_files[0].get(),
                                  0,
                                  &dex_container,
-                                 &error)) {
-    LOG(ERROR) << "Unable to process CompactDex file: " << error;
+                                 &error_msg)) {
+    LOG(ERROR) << "Unable to process CompactDex file: " << error_msg;
     return 1;
   }
 
@@ -132,9 +135,9 @@
                                       /*oat_dex_file*/ nullptr,
                                       /*verify*/ verify_output_file,
                                       /*verify_checksum*/ false,
-                                      &error);
+                                      &error_msg);
   if (new_dex_file == nullptr) {
-    LOG(ERROR) << "Unable to open dex file from memory: " << error;
+    LOG(ERROR) << "Unable to open dex file from memory: " << error_msg;
     return 1;
   }
 
@@ -251,3 +254,4 @@
 
   return art::DexlayoutDriver(argc, argv);
 }

@superr
Copy link
Author

superr commented Sep 11, 2018

Thanks @Furniel, it is getting closer :)

New error:

[ 72% 5928/8128] //art/dexlayout:compact_dex_converter link compact_dex_converte
FAILED: out/soong/.intermediates/art/dexlayout/compact_dex_converter/linux_glibc_x86_64/compact_dex_converter
prebuilts/clang/host/linux-x86/clang-r328903/bin/clang++  @out/soong/.intermediates/art/dexlayout/compact_dex_converter/linux_glibc_x86_64/compact_dex_converter.rsp -Wl,--start-group out/soong/.intermediates/external/compiler-rt/libcompiler_rt-extras/linux_glibc_x86_64_static/libcompiler_rt-extras.a -Wl,--end-group out/soong/.intermediates/system/core/base/libbase/linux_glibc_x86_64_shared/libbase.so out/soong/.intermediates/art/runtime/libart/linux_glibc_x86_64_shared/libart.so out/soong/.intermediates/art/dexlayout/libart-dexlayout/linux_glibc_x86_64_shared/libart-dexlayout.so out/soong/.intermediates/art/libdexfile/libdexfile/linux_glibc_x86_64_shared/libdexfile.so out/soong/.intermediates/external/libcxx/libc++/linux_glibc_x86_64_shared/libc++.so  -o out/soong/.intermediates/art/dexlayout/compact_dex_converter/linux_glibc_x86_64/compact_dex_converter -target x86_64-linux-gnu -Bprebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/bin -fuse-ld=lld -Wl,--no-undefined -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -Wl,--no-undefined-version --gcc-toolchain=prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8 --sysroot prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/sysroot -m64 -Bprebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/lib/gcc/x86_64-linux/4.8 -Lprebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/lib/gcc/x86_64-linux/4.8 -Lprebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/lib64 -ldl -lpthread -lm -lrt -Wl,-rpath,\$ORIGIN/../lib64 -Wl,-rpath,\$ORIGIN/lib64  -pie -nodefaultlibs -lgcc_s -lgcc -lc -lgcc_s -lgcc
/media/superr/Extra/superr/My_Stuff/Temp/Android/source_pie/prebuilts/clang/host/linux-x86/clang-r328903/bin/ld.lld: error: undefined symbol: art::InitLogging(char**, void (&)(char const*))
>>> referenced by compact_dex_converter_main.cc:184 (art/dexlayout/compact_dex_converter_main.cc:184)
>>>               out/soong/.intermediates/art/dexlayout/compact_dex_converter/linux_glibc_x86_64/obj/art/dexlayout/compact_dex_converter_main.o:(art::DexlayoutDriver(int, char**))

/media/superr/Extra/superr/My_Stuff/Temp/Android/source_pie/prebuilts/clang/host/linux-x86/clang-r328903/bin/ld.lld: error: undefined symbol: art::MemMap::Init()
>>> referenced by compact_dex_converter_main.cc:185 (art/dexlayout/compact_dex_converter_main.cc:185)
>>>               out/soong/.intermediates/art/dexlayout/compact_dex_converter/linux_glibc_x86_64/obj/art/dexlayout/compact_dex_converter_main.o:(art::DexlayoutDriver(int, char**))

/media/superr/Extra/superr/My_Stuff/Temp/Android/source_pie/prebuilts/clang/host/linux-x86/clang-r328903/bin/ld.lld: error: undefined symbol: art::OS::CreateEmptyFile(char const*)
>>> referenced by compact_dex_converter_main.cc:159 (art/dexlayout/compact_dex_converter_main.cc:159)
>>>               out/soong/.intermediates/art/dexlayout/compact_dex_converter/linux_glibc_x86_64/obj/art/dexlayout/compact_dex_converter_main.o:(art::DexlayoutDriver(int, char**))

/media/superr/Extra/superr/My_Stuff/Temp/Android/source_pie/prebuilts/clang/host/linux-x86/clang-r328903/bin/ld.lld: error: undefined symbol: unix_file::FdFile::WriteFully(void const*, unsigned long)
>>> referenced by compact_dex_converter_main.cc:165 (art/dexlayout/compact_dex_converter_main.cc:165)
>>>               out/soong/.intermediates/art/dexlayout/compact_dex_converter/linux_glibc_x86_64/obj/art/dexlayout/compact_dex_converter_main.o:(art::DexlayoutDriver(int, char**))

/media/superr/Extra/superr/My_Stuff/Temp/Android/source_pie/prebuilts/clang/host/linux-x86/clang-r328903/bin/ld.lld: error: undefined symbol: unix_file::FdFile::FlushCloseOrErase()
>>> referenced by compact_dex_converter_main.cc:169 (art/dexlayout/compact_dex_converter_main.cc:169)
>>>               out/soong/.intermediates/art/dexlayout/compact_dex_converter/linux_glibc_x86_64/obj/art/dexlayout/compact_dex_converter_main.o:(art::DexlayoutDriver(int, char**))
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
02:38:38 ninja failed with: exit status 1
build/make/core/main.mk:21: recipe for target 'run_soong_ui' failed
make: *** [run_soong_ui] Error 1

I appreciate the help :)

@Furniel
Copy link

Furniel commented Sep 11, 2018

@superr
Open Android.bp and find:

art_cc_binary {
    name: "compact_dex_converter",
    defaults: ["compact_dex_converter_defaults"],
    shared_libs: [
        "libart",
        "libart-dexlayout",
        "libdexfile",
    ],
}

art_cc_binary {
    name: "compact_dex_converterd",
    defaults: [
        "art_debug_defaults",
        "compact_dex_converter_defaults",
    ],
    shared_libs: [
        "libartd",
        "libartd-dexlayout",
        "libdexfiled",
    ],
}

and change to:

art_cc_binary {
    name: "compact_dex_converter",
    defaults: ["compact_dex_converter_defaults"],
    shared_libs: [
        "libartbase",
        "libart",
        "libart-dexlayout",
        "libdexfile",
    ],
}

art_cc_binary {
    name: "compact_dex_converterd",
    defaults: [
        "art_debug_defaults",
        "compact_dex_converter_defaults",
    ],
    shared_libs: [
        "libartbased",
        "libartd",
        "libartd-dexlayout",
        "libdexfiled",
    ],
}

@superr
Copy link
Author

superr commented Sep 11, 2018

Thanks @Furniel for all your help. Finally got it compiled :)

Unfortunately, it has the exact same issue as the ones you and @anestisb compiled.

$ bin/compact_dex_converter BasicDreams_classes.cdex 
Opened 'BasicDreams_classes.cdex', DEX version '001'
Illegal instruction (core dumped)

Any new ideas to get this converter working on older PC's? Thanks

@Furniel
Copy link

Furniel commented Sep 11, 2018

You can try to change compilation flags. But i'm not sure flags from what files used for host build. I recommend start from "/build/soog/cc/config" there you can find "x86_linux_host.go" open it and try to change

	// Extended cflags
	linuxX86Cflags = []string{
		"-msse3",
		"-mfpmath=sse",
		"-m32",
		"-march=prescott",
		"-D_FILE_OFFSET_BITS=64",
		"-D_LARGEFILE_SOURCE=1",
	}

you can try remove this

		"-march=prescott",
		"-msse3",
		"-mfpmath=sse",

if this won't help you can open "global.go" and change commonGlobalCflags, for example "-O2" to "-O1"

@superr
Copy link
Author

superr commented Sep 11, 2018

Thank you again @Furniel for your help :)

Sadly neither of those changes made a difference. I do not have the knowledge to troubleshoot this on my own. I understand if you do not wish to continue, but appreciate any additional help you may provide :)

@Furniel
Copy link

Furniel commented Sep 13, 2018

@superr
Ok, i don't know what cpu you use, but it seems that you can add -march and-mtune flags to linuxCflags in x86_linux_host.go. Set them to core2 and built new binary, maybe it will work. Have no more ideas how to fix this.
compact_dex_converter_core2.zip

@superr
Copy link
Author

superr commented Sep 14, 2018

Thanks @Furniel for the new binaries.
Unfortunately the same Illegal instruction.

CPU: AMD Phenom II X4 955

@Furniel
Copy link

Furniel commented Sep 15, 2018

@superr
Last try, built compact_dex_converter with amdfam10 optimization flags
compact_dex_converter_amdfam10.zip

@superr
Copy link
Author

superr commented Sep 16, 2018

Thank you once again @Furniel. I really appreciate all your time and effort :)

I wish I could bring good news after all this, but it's the same error again. I am able to work around it using valgrind as mentioned above. It is not ideal but it does seem to work. It will have to do for now, at least until an alternative cdex converter is created (hopefully one that can be compiled for Windows too).

@anestisb
Copy link
Owner

@superr I've managed to reproduce the issue on a dusty workstation and found the root cause of the problem. It's not the main build configs, it's an ART enforcement of SSE4.2 & POPCNT introduced in this commit. The converter tool is affected since it links with ART libraries.

Applying the following patch should resolve your problem:

diff --git a/build/Android.bp b/build/Android.bp
index 2a5598f..edb9eab 100644
--- a/build/Android.bp
+++ b/build/Android.bp
@@ -99,8 +99,8 @@ art_global_defaults {
                 // method InstructionSetFeatures::FromCppDefines(). Since invoking dex2oat directly
                 // does not pick up these flags, cross-compiling from a x86/x86_64 host to a
                 // x86/x86_64 target should not be affected.
-                "-msse4.2",
-                "-mpopcnt",
+                // "-msse4.2",
+                // "-mpopcnt",
             ],
         },
     },

I've also updated (see f099f29) the README links with some additional precompiled binaries for older CPUs. Hopefully, it will work for you too.

Needless to say that your workstation cannot be used anymore to pre-optimise apps when generating system images. dex2oat will fail since it expects an SSE4.2 compatible host for the activated ISA features. You might be able to workaround it by disabling some default ISA features from within ART defaults, although too much hassle for 12+ year old CPUs. Probably its time to update your setup :)

@superr
Copy link
Author

superr commented Sep 28, 2018

The new binaries for older CPU worked perfectly! Thank you for all your extra effort on this old PC :)

This PC is still plenty fast for most things I do on it so I have not been motivated to upgrade. However, this is the second time exactly this missing SSE4.2 has caused an issue. Trying to run a certain OS as a VM failed for the same reason. I suppose it is time to start shopping. Thanks again for your help. You are very much appreciated :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants