Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

7.0.0 rc1: 16 libc++ tests failing on macOS/x86_64 (Exit Code: -11) #37821

Closed
vedantk opened this issue Aug 7, 2018 · 20 comments
Closed

7.0.0 rc1: 16 libc++ tests failing on macOS/x86_64 (Exit Code: -11) #37821

vedantk opened this issue Aug 7, 2018 · 20 comments
Assignees
Labels
bugzilla Issues migrated from bugzilla libc++ Bugs in the libc++ C++ Standard Library. Not GNU libstdc++. Not libc++abi.

Comments

@vedantk
Copy link
Collaborator

vedantk commented Aug 7, 2018

Bugzilla Link 38473
Resolution FIXED
Resolved on Sep 10, 2020 10:26
Version 7.0
OS All
CC @zmodem,@ldionne,@mclow

Extended Description

These tests all fail in what looks to be the same way:

libc++ :: std/language.support/support.exception/except.nested/assign.pass.cpp
libc++ :: std/language.support/support.exception/except.nested/ctor_copy.pass.cpp
libc++ :: std/language.support/support.exception/except.nested/ctor_default.pass.cpp
libc++ :: std/language.support/support.exception/except.nested/rethrow_if_nested.pass.cpp
libc++ :: std/language.support/support.exception/except.nested/rethrow_nested.pass.cpp
libc++ :: std/language.support/support.exception/propagation/make_exception_ptr.pass.cpp
libc++ :: std/language.support/support.exception/propagation/rethrow_exception.pass.cpp
libc++ :: std/thread/futures/futures.async/async.pass.cpp
libc++ :: std/thread/futures/futures.promise/dtor.pass.cpp
libc++ :: std/thread/futures/futures.promise/set_exception.pass.cpp
libc++ :: std/thread/futures/futures.promise/set_exception_at_thread_exit.pass.cpp
libc++ :: std/thread/futures/futures.shared_future/get.pass.cpp
libc++ :: std/thread/futures/futures.task/futures.task.members/dtor.pass.cpp
libc++ :: std/thread/futures/futures.task/futures.task.members/make_ready_at_thread_exit.pass.cpp
libc++ :: std/thread/futures/futures.task/futures.task.members/operator.pass.cpp
libc++ :: std/thread/futures/futures.unique_future/get.pass.cpp

To reproduce the failure, I did:

$ export MACOSX_DEPLOYMENT_TARGET="10.9"
$ ./llvm/utils/release/test-release.sh -release 7.0.0 -rc 1 -triple x86_64-apple-darwin -no-compare-files -j20

Here's a sample failure:

FAIL: libc++ :: std/language.support/support.exception/except.nested/assign.pass.cpp (49834 of 54980)
******************** TEST 'libc++ :: std/language.support/support.exception/except.nested/assign.pass.cpp' FAILED ********************
Compiled With: ['/Users/vsk/src/llvm.org-master/rc1/Phase2/Release/llvmCore-7.0.0-rc1.install/usr/local/bin/clang++', '-o', '/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.o', '-x', 'c++', '/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/std/language.support/support.exception/except.nested/assign.pass.cpp', '-c', '-v', '-arch', 'x86_64', '-mmacosx-version-min=10.13', '-D_LIBCPP_DISABLE_AVAILABILITY', '-ftemplate-depth=270', '-Werror=thread-safety', '-std=c++2a', '-include', '/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/support/nasty_macros.hpp', '-nostdinc++', '-I/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/include', '-I/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/include/c++build', '-isysroot', '/Volumes/Xcode10A230a_m18A350_i16A343_t16J338_w16R338a_b16P349_XcodeInternals_32bitSupport_FastSim_Boost_ASan_36GB/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk', '-I/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/support', '-DLIBCXX_FILESYSTEM_STATIC_TEST_ROOT="/Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/std/input.output/filesystems/Inputs/static_test_env"', '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_ROOT="/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/filesystem/Output/dynamic_env"', '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_HELPER="/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python /Users/vsk/src/llvm.org-master/rc1/llvm.src/projects/libcxx/test/support/filesystem_dynamic_test_helper.py"', '-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER', '-Wall', '-Wextra', '-Werror', '-Wuser-defined-warnings', '-Wshadow', '-Wno-unused-command-line-argument', '-Wno-attributes', '-Wno-pessimizing-move', '-Wno-c++11-extensions', '-Wno-user-defined-literals', '-Wno-noexcept-type', '-Wno-aligned-allocation-unavailable', '-Wsign-compare', '-Wunused-variable', '-Wunused-parameter', '-Wunreachable-code', '-Wno-conversion', '-Wno-unused-local-typedef', '-c', '&&', '/Users/vsk/src/llvm.org-master/rc1/Phase2/Release/llvmCore-7.0.0-rc1.install/usr/local/bin/clang++', '-o', '/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.exe', '/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.o', '-v', '-arch', 'x86_64', '-mmacosx-version-min=10.13', '-D_LIBCPP_DISABLE_AVAILABILITY', '-ftemplate-depth=270', '-L/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/./lib', '-Wl,-rpath,/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/./lib', '-nodefaultlibs', '-lc++experimental', '-lc++fs', '-lc++', '-lSystem']
Command: ['/Users/vsk/src/llvm.org-master/rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.exe']
Exit Code: -11

In this case, assign.pass.cpp.exe does not exist. The directory it's supposed to be in exists and is empty.

If I strip out the single-quotes, perform the compile+link steps manually, and run assign.pass.cpp.exe it passes.

@vedantk
Copy link
Collaborator Author

vedantk commented Aug 7, 2018

assigned to @ldionne

@mclow
Copy link
Contributor

mclow commented Aug 13, 2018

I'll be looking at this this morning.

@mclow
Copy link
Contributor

mclow commented Aug 14, 2018

Odd that even though you exported MACOSX_DEPLOYMENT_TARGET="10.9", the compile command contains '-mmacosx-version-min=10.13'.

(Thanks to Louis for noticing this)

@mclow
Copy link
Contributor

mclow commented Aug 15, 2018

There's clearly multiple things going on here.

  1. Error -11 is a segmentation fault.
  2. The reason you can't find the programs is that lit removes them after they've been run. They were built successfully.
  3. When you build them manually, and then run them, you are running against the system dylib, so you don't see the failure.
  4. The programs are definitely crashing. (see below).
  5. The programs are being built with a macosx target of 10.13, not 10.9

Here's how you can build (and run) just one test.

$ 'rc1/Phase2/Release/llvmCore-7.0.0-rc1.install/usr/local/bin/clang++' '-o' 'rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/rethrow_nested.pass.cpp.o' '-x' 'c++' 'rc1/llvm.src/projects/libcxx/test/std/language.support/support.exception/except.nested/rethrow_nested.pass.cpp' '-c' '-v' '-arch' 'x86_64' '-mmacosx-version-min=10.9' '-D_LIBCPP_DISABLE_AVAILABILITY' '-ftemplate-depth=270' '-Werror=thread-safety' '-std=c++17' '-include' 'rc1/llvm.src/projects/libcxx/test/support/nasty_macros.hpp' '-nostdinc++' '-Irc1/llvm.src/projects/libcxx/include' '-Irc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/include/c++build' '-isysroot' '/Library/Developer/CommandLineTools/SDKs/MacOSX10.13.sdk' '-Irc1/llvm.src/projects/libcxx/test/support' '-DLIBCXX_FILESYSTEM_STATIC_TEST_ROOT="rc1/llvm.src/projects/libcxx/test/std/input.output/filesystems/Inputs/static_test_env"' '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_ROOT="rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/filesystem/Output/dynamic_env"' '-DLIBCXX_FILESYSTEM_DYNAMIC_TEST_HELPER="/usr/bin/python rc1/llvm.src/projects/libcxx/test/support/filesystem_dynamic_test_helper.py"' '-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER' '-Wall' '-Wextra' '-Werror' '-Wuser-defined-warnings' '-Wshadow' '-Wno-unused-command-line-argument' '-Wno-attributes' '-Wno-pessimizing-move' '-Wno-c++11-extensions' '-Wno-user-defined-literals' '-Wno-noexcept-type' '-Wno-aligned-allocation-unavailable' '-Wsign-compare' '-Wunused-variable' '-Wunused-parameter' '-Wunreachable-code' '-Wno-conversion' '-Wno-unused-local-typedef' '-c'

$ 'rc1/Phase2/Release/llvmCore-7.0.0-rc1.install/usr/local/bin/clang++' '-o' 'rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/rethrow_nested.pass.cpp.exe' 'rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/rethrow_nested.pass.cpp.o' '-v' '-arch' 'x86_64' '-mmacosx-version-min=10.9' '-D_LIBCPP_DISABLE_AVAILABILITY' '-ftemplate-depth=270' '-Lrc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/./lib' '-Wl,-rpath,rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/./lib' '-nodefaultlibs' '-lc++experimental' '-lc++fs' '-lc++' '-lSystem'

$ DYLD_LIBRARY_PATH=rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/lib rc1/Phase3/Release/llvmCore-7.0.0-rc1.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/rethrow_nested.pass.cpp.exe

Segmentation fault: 11

I suspect that some of the other bits - (like libcxxabi) are not built with the same options as libcxx and/or the test programs and that is causing the crashes.

@vedantk
Copy link
Collaborator Author

vedantk commented Aug 24, 2018

Thanks Marshall, I can reproduce the crash when specifying the correct DYLD_LIBRARY_PATH:

  • thread #​1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #​0: 0x00000001001b00cd libunwind.dylib_Unwind_RaiseException + 93 libunwind.dylib_Unwind_RaiseException:
    -> 0x1001b00cd <+93>: movaps %xmm0, 0x10(%r14)
    0x1001b00d2 <+98>: leaq -0x398(%rbp), %rbx
    0x1001b00d9 <+105>: movq %rbx, %rdi
    0x1001b00dc <+108>: movq %r15, %rsi
    Target 0: (assign.pass.cpp.exe) stopped.
    (lldb) bt
  • thread #​1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    • frame #​0: 0x00000001001b00cd libunwind.dylib_Unwind_RaiseException + 93 frame #&#8203;1: 0x0000000100190e9f libc++abi.dylib__cxa_rethrow_primary_exception + 239
      frame #​2: 0x00000001000afebc libc++.1.dylibstd::rethrow_exception(std::exception_ptr) + 12 frame #&#8203;3: 0x0000000100000ab8 assign.pass.cpp.exemain at assign.pass.cpp:53
      frame #​4: 0x00007fff57147015 libdyld.dylib`start + 1
      (lldb) image list
      [ 0] 8D2E4DC3-765E-3C89-A451-23AA36AFB082 0x0000000100000000 /Users/vsk/src/llvm.org-master/rc2/Phase3/Release/llvmCore-7.0.0-rc2.obj/projects/libcxx/test/std/language.support/support.exception/except.nested/Output/assign.pass.cpp.exe
      [ 1] 8A72DE9C-A136-3506-AA02-4BA2B82DCAF3 0x0000000100003000 /usr/lib/dyld
      [ 2] 053A1AC0-ED15-38D8-8B05-174BDAEC9B72 0x00000001000a1000 /Users/vsk/src/llvm.org-master/rc2/Phase3/Release/llvmCore-7.0.0-rc2.obj/lib/libc++.1.dylib
      [ 3] CD555F3B-FDDB-35E5-A2FB-FBBF3D62031A 0x00007fff54e60000 /usr/lib/libSystem.B.dylib
      [ 4] 20AF2CAA-9E97-3FB5-9BCE-A4E793429348 0x0000000100173000 /Users/vsk/src/llvm.org-master/rc2/Phase3/Release/llvmCore-7.0.0-rc2.obj/lib/libc++abi.dylib
      ...
      [ 37] 1E105381-88F9-3296-813F-A1E9B14BD93C 0x00000001001a8000 /Users/vsk/src/llvm.org-master/rc2/Phase3/Release/llvmCore-7.0.0-rc2.obj/lib/libunwind.dylib

@vedantk
Copy link
Collaborator Author

vedantk commented Aug 24, 2018

Could the GPFLT be due to an alignment exception?

The crashing instruction appears to be:

0x1001b00cd <+93>: movaps %xmm0, 0x10(%r14)

I think that requires 16 byte alignment, but 0x10(%r14) isn't sufficiently aligned?

(lldb) reg read xmm0
xmm0 = {0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00}
(lldb) p/x $r14+0x10
(unsigned long) $2 = 0x00000001003002e8

@ldionne
Copy link
Member

ldionne commented Aug 24, 2018

I'm starting to suspect this has something to do with the changes we did to disable the use of aligned allocation functions when the deployment target does not support them. Maybe there's something wrong with that change. I'll try to reproduce using ./llvm/utils/release/test-release.sh -release 7.0.0 -rc 1 -triple x86_64-apple-darwin -no-compare-files and report when I'm able to repro.

@ldionne
Copy link
Member

ldionne commented Aug 27, 2018

Minimal reproduction script
This has nothing to do with the bootstrapping process. I can reproduce with a system compiler and the RC1 sources.

@ldionne
Copy link
Member

ldionne commented Aug 27, 2018

I can now also reproduce on ToT, but one has to include libunwind in the set of built projects, which I was not doing initially (using the monorepo setup).

@ldionne
Copy link
Member

ldionne commented Aug 28, 2018

Updated reproduction script
Okay, so with Marshall's help, I was able to narrow it down to a single revision on libunwind:

r291830 (around Release 5) => SEGFAULT
r276215 => SEGFAULT
r276214 => no SEGFAULT

In other words, r276215 introduces the failure. I also attached the script I used to reproduce. My setup was a checkout of LLVM trunk with projects/libcxx on trunk, projects/libcxxabi on trunk, and projects/libunwind on the various revisions shown above.

@ldionne
Copy link
Member

ldionne commented Aug 28, 2018

This has been failing for a long time, but I don't know why we only caught it recently. It is not impossible that we were picking up the system's libunwind, which may be older than that revision of libunwind.

@ldionne
Copy link
Member

ldionne commented Aug 28, 2018

It's possible to reproduce on Trunk with our current lit test suite. It suffices to make sure that libunwind is being built ninja -C build unwind to trigger the issue.

Also note that configuring CMake with -DLIBCXXABI_USE_LLVM_UNWINDER=ON seems to solve the problem (that option defaults to OFF).

@ldionne
Copy link
Member

ldionne commented Aug 28, 2018

With -DLIBCXXABI_USE_LLVM_UNWINDER=OFF, the command line used to build cxa_exception.cpp is:

$CXX  -D_LIBCPP_DISABLE_EXTERN_TEMPLATE -D_LIBCPP_ENABLE_CXX17_REMOVED_UNEXPECTED_FUNCTIONS -D_LIBCXXABI_BUILDING_LIBRARY -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS 
-Iprojects/libcxxabi/src 
-I/Users/ldionne/work/llvm/libcxxabi/src 
-I$SDKROOT/usr/include/libxml2 
-Iinclude 
-I/Users/ldionne/work/llvm/llvm/include 
-I/Users/ldionne/work/llvm/libcxxabi/include 
-I/Users/ldionne/work/llvm/libcxx/include 
-fPIC -fvisibility-inlines-hidden -std=c++11 -O2 -g -DNDEBUG -isysroot $SDKROOT -mmacosx-version-min=10.XX -fPIC -nostdinc++ -fstrict-aliasing -funwind-tables -D_DEBUG -std=c++11 -c /Users/ldionne/work/llvm/libcxxabi/src/cxa_exception.cpp

and unwind.h is: $SDKROOT/usr/include/unwind.h

With -DLIBCXXABI_USE_LLVM_UNWINDER=ON, the command line is:

$CXX -DLIBCXXABI_USE_LLVM_UNWINDER -D_LIBCPP_DISABLE_EXTERN_TEMPLATE -D_LIBCPP_ENABLE_CXX17_REMOVED_UNEXPECTED_FUNCTIONS -D_LIBCXXABI_BUILDING_LIBRARY -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS 
-Iprojects/libcxxabi/src 
-I/Users/ldionne/work/llvm/libcxxabi/src 
-I$SDKROOT/usr/include/libxml2 
-Iinclude 
-I/Users/ldionne/work/llvm/llvm/include 
-I/Users/ldionne/work/llvm/libcxxabi/include 
-I/Users/ldionne/work/llvm/libunwind/include 
-I/Users/ldionne/work/llvm/libcxx/include 
-fPIC -fvisibility-inlines-hidden -std=c++11 -O2 -g -DNDEBUG -isysroot $SDKROOT -mmacosx-version-min=10.XX -fPIC   -nostdinc++ -fstrict-aliasing -funwind-tables -D_DEBUG -std=c++11 -c /Users/ldionne/work/llvm/libcxxabi/src/cxa_exception.cpp

and unwind.h is /Users/ldionne/work/llvm/libunwind/include/unwind.h. Notice how -DLIBCXXABI_USE_LLVM_UNWINDER=ON causes -I/Users/ldionne/work/llvm/libunwind/include to be added to the header search paths.

In other words, without -DLIBCXXABI_USE_LLVM_UNWINDER=ON, we're using the system's unwind.h, to build libc++abi, and so we're using struct _Unwind_Exception provided by the system's unwind.h, which has 8 bit alignment (because it predates r276215, where the alignment was changed). However, when we link against LLVM's libunwind.dylib (using DYLD_LIBRARY_PATH as in the reproduction), we're linking against a libunwind.dylib that thinks the alignment of struct _Unwind_Exception is 16 bits, because it was built with the new headers.

This is where we get a mismatch: libc++abi thinks the alignment of struct _Unwind_Exception is 8, but libunwind.dylib think it is 16.

As soon as we go back to using the system's libunwind.dylib (which also thinks alignment is 8), the problem goes away. Similarly, when we start building libc++abi.dylib with the headers that know the alignment to be 16, the problem goes away.

@ldionne
Copy link
Member

ldionne commented Aug 28, 2018

I think we understand the problem pretty well at this point. Marshall and I have discussed the following solutions:

  1. When building libunwind, use the target system's unwind.h headers. This means that we'll build and ship a libunwind.dylib that's compatible with the headers present on the system. This also means that libc++abi will have to be built against those system headers, otherwise we'll run into the issue encountered here, but reversed (libc++abi will think 16 bytes alignment, but libunwind will think 8 bytes, which is technically OK but let's not go there). The downside of this approach is that we're not shipping a release that's self consistent: the unwind.h header shipped with LLVM 7 should never be used, because the libunwind.dylib will have been built with different and incompatible headers.

  2. Build libc++abi AND libunwind against the LLVM 7 unwind.h headers. This means that we'd ship libc++.dylib/libc++abi.dylib/libunwind.dylib that are consistent with the headers provided with LLVM 7, but not with the system's headers. The downside is that a program using the system's unwind.h headers could NOT be linked against libc++abi.dylib/libunwind.dylib shipped with LLVM 7, because those would have been built with newer and incompatible libunwind headers.

We currently lean towards option (1) for lack of a better solution.

@zmodem
Copy link
Collaborator

zmodem commented Sep 4, 2018

We currently lean towards option (1) for lack of a better solution.

Is there a patch or something for this?

@ldionne
Copy link
Member

ldionne commented Sep 4, 2018

Not yet. I've been distracted by other work and you mentioned this was not a release blocker for the moment, so I set this aside. Please LMK if you feel like this needs to be resolved for the release and I'll try to re-prioritize.

@zmodem
Copy link
Collaborator

zmodem commented Sep 5, 2018

Not yet. I've been distracted by other work and you mentioned this was not a
release blocker for the moment, so I set this aside. Please LMK if you feel
like this needs to be resolved for the release and I'll try to re-prioritize.

Right, let's not wait for this.

@ldionne
Copy link
Member

ldionne commented Sep 10, 2020

Since then, Apple has updated the version of libunwind (both the headers and the dylib) shipped in SDKs. I don't think this issue is still relevant. Vedant, please re-open if you disagree.

@llvmbot
Copy link
Collaborator

llvmbot commented Nov 27, 2021

mentioned in issue llvm/llvm-bugzilla-archive#39051

1 similar comment
@llvmbot
Copy link
Collaborator

llvmbot commented Nov 27, 2021

mentioned in issue llvm/llvm-bugzilla-archive#39051

@llvmbot llvmbot transferred this issue from llvm/llvm-bugzilla-archive Dec 10, 2021
dyung added a commit that referenced this issue Sep 23, 2022
This reverts commit 3f08d24.

The commit this change is fixing is being reverted due to GHI #57796 and #37821, so revert this commit as well.
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bugzilla Issues migrated from bugzilla libc++ Bugs in the libc++ C++ Standard Library. Not GNU libstdc++. Not libc++abi.
Projects
None yet
Development

No branches or pull requests

5 participants