Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Building julia-debug with TSAN on julia-release-1.10 fails #51774

Open
Liozou opened this issue Oct 19, 2023 · 5 comments
Open

Building julia-debug with TSAN on julia-release-1.10 fails #51774

Liozou opened this issue Oct 19, 2023 · 5 comments
Labels
domain:building Build system, or building Julia or its dependencies

Comments

@Liozou
Copy link
Member

Liozou commented Oct 19, 2023

I am trying to make a TSAN build of Julia to investigate a race condition, but the build is failing. I am on the backport-release-1.10 branch at the current head (commit 110e9ba) with a fresh git clone of the repo. I incorporated 2e17bb5 as well but it makes no difference.

I added the following lines to contrib/tsan/Make.user.tsan (although the TSAN_OPTIONS is probably not meant to go there):

TSAN_OPTIONS=second_deadlock_stack=1

override JULIA_BUILD_MODE=debug

FORCE_ASSERTIONS=1
LLVM_ASSERTIONS=1

and I get the following output: build_output.txt

The TSAN warning itself may be a dup of #46178?

@vtjnash
Copy link
Sponsor Member

vtjnash commented Oct 19, 2023

You will need to override the exitcode in the options also to allow the build to finish, despite running into #46178

@Liozou
Copy link
Member Author

Liozou commented Oct 19, 2023

Thanks, so basically changing the false to true after the corresponding error in sysimage.mk (line 72, after the "This error might be fixed by..." echo)?
Relaunching the build script also made the following warning reappear (it was already hidden there in the first log):

WARNING: using mismatched version for sanitizers:
  want 
  To resolve this warning, you could try either of the following suggestions: 
  1. Run the following command: make -C deps uninstall
  2. Remove the following directory: /home/liozou/julia-1.10-tsan/usr

Since the repo is a fresh git clone, I assume there may be something wrong there as well. The empty want line is quite suspicious.

@Liozou
Copy link
Member Author

Liozou commented Oct 19, 2023

So I did that falsetrue change, and I also retried by passing a -k flag to the make commands of contrib/tsan/build.sh but it still fails (although later). The beginning of the log is the same as before and it then continues like so:

Start of the log
    JULIA tsan-build/tsan/usr/lib/julia/sys-debug-o.a
Collecting and executing precompile statements
└ Collect (Basic: /usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x23
/usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x23
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=2157108)
  Cycle in lock order graph: M0 (0x7b1000025458) => M1 (0x7b1000480758) => M0

  Mutex M1 acquired here while holding mutex M0 in main thread:
    #0 pthread_mutex_lock /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_common_interceptors.inc:4457 (julia-debug+0x6fac8)
    #1 __gthread_mutex_lock(pthread_mutex_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x60913)
    #2 __gthread_recursive_mutex_lock(pthread_mutex_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xf1b15)
    #3 std::recursive_mutex::lock() :? (libjulia-codegen-debug.so.1.10+0x18b535)
    #4 std::unique_lock<std::recursive_mutex>::lock() :? (libjulia-codegen-debug.so.1.10+0x18b4db)
    #5 std::unique_lock<std::recursive_mutex>::unique_lock(std::recursive_mutex&) :? (libjulia-codegen-debug.so.1.10+0x18b39e)
    #6 llvm::orc::ThreadSafeContext::Lock::Lock(std::shared_ptr<llvm::orc::ThreadSafeContext::State>) :? (libjulia-codegen-debug.so.1.10+0x18b0d5)
    #7 llvm::orc::ThreadSafeContext::getLock() const :? (libjulia-codegen-debug.so.1.10+0x18af26)
    #8 jl_codegen_params_t::jl_codegen_params_t(llvm::orc::ThreadSafeContext, llvm::DataLayout, llvm::Triple) :? (libjulia-codegen-debug.so.1.10+0x1b08d9)
    #9 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long, llvm::orc::ThreadSafeContext, bool) jitlayers.cpp:? (libjulia-codegen-debug.so.1.10+0x197d73)
    #10 jl_generate_fptr_impl ??:? (libjulia-codegen-debug.so.1.10+0x19789d)
    #11 jl_compile_method_internal ??:? (libjulia-internal-debug.so.1.10+0x5fba1)
    #12 gen_cfun_wrapper(llvm::Module*, jl_codegen_params_t&, function_sig_t const&, _jl_value_t*, char const*, _jl_value_t*, _jl_method_instance_t*, jl_unionall_t*, jl_svec_t*, jl_array_t**) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x499e7)
    #13 emit_cfunction(jl_codectx_t&, _jl_value_t*, jl_cgval_t const&, _jl_value_t*, jl_svec_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x8ed13)
    #14 emit_expr(jl_codectx_t&, _jl_value_t*, long) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x7bf44)
    #15 emit_ssaval_assign(jl_codectx_t&, long, _jl_value_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xf00f5)
    #16 emit_stmtpos(jl_codectx_t&, _jl_value_t*, int) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xeca03)
    #17 emit_function(llvm::orc::ThreadSafeModule&, _jl_method_instance_t*, _jl_code_info_t*, _jl_value_t*, jl_codegen_params_t&) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x58240)
    #18 jl_emit_code(llvm::orc::ThreadSafeModule&, _jl_method_instance_t*, _jl_code_info_t*, _jl_value_t*, jl_codegen_params_t&) :? (libjulia-codegen-debug.so.1.10+0x4ea72)
    #19 jl_emit_codeinst(llvm::orc::ThreadSafeModule&, _jl_code_instance_t*, _jl_code_info_t*, jl_codegen_params_t&) :? (libjulia-codegen-debug.so.1.10+0x5b0bf)
    #20 jl_compile_workqueue(jl_codegen_params_t&, llvm::Module&, CompilationPolicy) :? (libjulia-codegen-debug.so.1.10+0x5c5eb)
    #21 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long, llvm::orc::ThreadSafeContext, bool) jitlayers.cpp:? (libjulia-codegen-debug.so.1.10+0x197ff7)
    #22 jl_generate_fptr_impl ??:? (libjulia-codegen-debug.so.1.10+0x19789d)
    #23 jl_compile_method_internal ??:? (libjulia-internal-debug.so.1.10+0x5fba1)
    #24 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65a50)
    #25 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #26 <null> <null> (0x7fc6237021d1)
    #27 <null> <null> (0x7fc621904e42)
    #28 <null> <null> (0x7fc62190dbcd)
    #29 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65aef)
    #30 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #31 jl_apply interpreter.c:? (libjulia-internal-debug.so.1.10+0xaae5c)
    #32 do_call interpreter.c:? (libjulia-internal-debug.so.1.10+0xaa9d9)
    #33 eval_value interpreter.c:? (libjulia-internal-debug.so.1.10+0xa7b73)
    #34 eval_stmt_value interpreter.c:? (libjulia-internal-debug.so.1.10+0xa9c45)
    #35 eval_body interpreter.c:? (libjulia-internal-debug.so.1.10+0xa5d3e)
    #36 jl_interpret_toplevel_thunk :? (libjulia-internal-debug.so.1.10+0xa6e0d)
    #37 jl_toplevel_eval_flex :? (libjulia-internal-debug.so.1.10+0xec1b4)
    #38 ijl_toplevel_eval ??:? (libjulia-internal-debug.so.1.10+0xeebbc)
    #39 ijl_toplevel_eval_in ??:? (libjulia-internal-debug.so.1.10+0xef01c)
    #40 <null> <null> (0x7fc62770917a)
    #41 <null> <null> (0x7fc62770925c)
    #42 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65aef)
    #43 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #44 jl_apply interpreter.c:? (libjulia-internal-debug.so.1.10+0xaae5c)
    #45 do_call interpreter.c:? (libjulia-internal-debug.so.1.10+0xaa9d9)
    #46 eval_value interpreter.c:? (libjulia-internal-debug.so.1.10+0xa7b73)
    #47 eval_stmt_value interpreter.c:? (libjulia-internal-debug.so.1.10+0xa9c45)
    #48 eval_body interpreter.c:? (libjulia-internal-debug.so.1.10+0xa5d3e)
    #49 jl_interpret_toplevel_thunk :? (libjulia-internal-debug.so.1.10+0xa6e0d)
    #50 jl_toplevel_eval_flex :? (libjulia-internal-debug.so.1.10+0xec1b4)
    #51 jl_toplevel_eval_flex :? (libjulia-internal-debug.so.1.10+0xeb868)
    #52 ijl_toplevel_eval ??:? (libjulia-internal-debug.so.1.10+0xeebbc)
    #53 ijl_toplevel_eval_in ??:? (libjulia-internal-debug.so.1.10+0xef01c)
    #54 <null> <null> (0x7fc628702160)
    #55 jl_fptr_args :? (libjulia-internal-debug.so.1.10+0x52046)
    #56 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65985)
    #57 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #58 <null> <null> (0x7fc62890d283)
    #59 <null> <null> (0x7fc62811e1bf)
    #60 <null> <null> (0x7fc62811e2ac)
    #61 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65aef)
    #62 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #63 <null> <null> (0x7fc62817436d)
    #64 <null> <null> (0x7fc628144e47)
    #65 <null> <null> (0x7fc62814403d)
    #66 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65aef)
    #67 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #68 jl_apply jlapi.c:? (libjulia-internal-debug.so.1.10+0x15457c)
    #69 true_main jlapi.c:? (libjulia-internal-debug.so.1.10+0x156a28)
    #70 jl_repl_entrypoint ??:? (libjulia-internal-debug.so.1.10+0x1567df)
    #71 jl_load_repl ??:? (libjulia-debug.so.1.10+0x13693)
    #72 main ??:? (julia-debug+0xe8ddd)

    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message

  Mutex M0 acquired here while holding mutex M1 in thread T6:
    #0 pthread_mutex_lock /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_common_interceptors.inc:4457 (julia-debug+0x6fac8)
    #1 __gthread_mutex_lock(pthread_mutex_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x60913)
    #2 __gthread_recursive_mutex_lock(pthread_mutex_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xf1b15)
    #3 std::recursive_mutex::lock() :? (libjulia-codegen-debug.so.1.10+0x18b535)
    #4 std::unique_lock<std::recursive_mutex>::lock() :? (libjulia-codegen-debug.so.1.10+0x18b4db)
    #5 std::unique_lock<std::recursive_mutex>::unique_lock(std::recursive_mutex&) :? (libjulia-codegen-debug.so.1.10+0x18b39e)
    #6 llvm::orc::ThreadSafeContext::Lock::Lock(std::shared_ptr<llvm::orc::ThreadSafeContext::State>) :? (libjulia-codegen-debug.so.1.10+0x18b0d5)
    #7 llvm::orc::ThreadSafeContext::getLock() const :? (libjulia-codegen-debug.so.1.10+0x18af26)
    #8 jl_codegen_params_t::jl_codegen_params_t(llvm::orc::ThreadSafeContext, llvm::DataLayout, llvm::Triple) :? (libjulia-codegen-debug.so.1.10+0x1b08d9)
    #9 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long, llvm::orc::ThreadSafeContext, bool) jitlayers.cpp:? (libjulia-codegen-debug.so.1.10+0x197d73)
    #10 jl_generate_fptr_impl ??:? (libjulia-codegen-debug.so.1.10+0x19789d)
    #11 jl_compile_method_internal ??:? (libjulia-internal-debug.so.1.10+0x5fba1)
    #12 gen_cfun_wrapper(llvm::Module*, jl_codegen_params_t&, function_sig_t const&, _jl_value_t*, char const*, _jl_value_t*, _jl_method_instance_t*, jl_unionall_t*, jl_svec_t*, jl_array_t**) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x499e7)
    #13 emit_cfunction(jl_codectx_t&, _jl_value_t*, jl_cgval_t const&, _jl_value_t*, jl_svec_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x8ed13)
    #14 emit_expr(jl_codectx_t&, _jl_value_t*, long) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x7bf44)
    #15 emit_ssaval_assign(jl_codectx_t&, long, _jl_value_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xf00f5)
    #16 emit_stmtpos(jl_codectx_t&, _jl_value_t*, int) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xeca03)
    #17 emit_function(llvm::orc::ThreadSafeModule&, _jl_method_instance_t*, _jl_code_info_t*, _jl_value_t*, jl_codegen_params_t&) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x58240)
    #18 jl_emit_code(llvm::orc::ThreadSafeModule&, _jl_method_instance_t*, _jl_code_info_t*, _jl_value_t*, jl_codegen_params_t&) :? (libjulia-codegen-debug.so.1.10+0x4ea72)
    #19 jl_emit_codeinst(llvm::orc::ThreadSafeModule&, _jl_code_instance_t*, _jl_code_info_t*, jl_codegen_params_t&) :? (libjulia-codegen-debug.so.1.10+0x5b0bf)
    #20 jl_compile_workqueue(jl_codegen_params_t&, llvm::Module&, CompilationPolicy) :? (libjulia-codegen-debug.so.1.10+0x5c5eb)
    #21 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long, llvm::orc::ThreadSafeContext, bool) jitlayers.cpp:? (libjulia-codegen-debug.so.1.10+0x197ff7)
    #22 jl_generate_fptr_impl ??:? (libjulia-codegen-debug.so.1.10+0x19789d)
    #23 jl_compile_method_internal ??:? (libjulia-internal-debug.so.1.10+0x5fba1)
    #24 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65a50)
    #25 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #26 jl_apply task.c:? (libjulia-internal-debug.so.1.10+0xb2a5c)
    #27 start_task task.c:? (libjulia-internal-debug.so.1.10+0xb6f15)

  Thread T6 (tid=0, running) created by thread T4 at:
    #0 ijl_new_task ??:? (libjulia-internal-debug.so.1.10+0xb5aa0)
    #1 <null> <null> (0x7fc611607938)
    #2 <null> <null> (0x7fc6116028bc)
    #3 <null> <null> (0x7fc6118021a9)
    #4 <null> <null> (0x7fc6118022b5)
    #5 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65aef)
    #6 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #7 jl_apply task.c:? (libjulia-internal-debug.so.1.10+0xb2a5c)
    #8 start_task task.c:? (libjulia-internal-debug.so.1.10+0xb6f15)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) codegen.cpp:? in __gthread_mutex_lock(pthread_mutex_t*)
==================
└ Collect (Basic: ◓ , REPL 0/0: ◓ 204) => Execute ◓ 39/usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x23
└ Collect (Basic: ◐ , REPL 0/0: ◐ 204) => Execute ◐ 39/usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x23
└ Collect (Basic: ◒ , REPL 0/0: ◒ 204) => Execute ◒ 39==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=2159594)
  Cycle in lock order graph: M0 (0x7b1000025458) => M1 (0x7b100005f418) => M0

  Mutex M1 acquired here while holding mutex M0 in main thread:
    #0 pthread_mutex_lock /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_common_interceptors.inc:4457 (julia-debug+0x6fac8)
    #1 __gthread_mutex_lock(pthread_mutex_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x60913)
    #2 __gthread_recursive_mutex_lock(pthread_mutex_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xf1b15)
    #3 std::recursive_mutex::lock() :? (libjulia-codegen-debug.so.1.10+0x18b535)
    #4 std::unique_lock<std::recursive_mutex>::lock() :? (libjulia-codegen-debug.so.1.10+0x18b4db)
    #5 std::unique_lock<std::recursive_mutex>::unique_lock(std::recursive_mutex&) :? (libjulia-codegen-debug.so.1.10+0x18b39e)
    #6 llvm::orc::ThreadSafeContext::Lock::Lock(std::shared_ptr<llvm::orc::ThreadSafeContext::State>) :? (libjulia-codegen-debug.so.1.10+0x18b0d5)
    #7 llvm::orc::ThreadSafeContext::getLock() const :? (libjulia-codegen-debug.so.1.10+0x18af26)
    #8 jl_codegen_params_t::jl_codegen_params_t(llvm::orc::ThreadSafeContext, llvm::DataLayout, llvm::Triple) :? (libjulia-codegen-debug.so.1.10+0x1b08d9)
    #9 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long, llvm::orc::ThreadSafeContext, bool) jitlayers.cpp:? (libjulia-codegen-debug.so.1.10+0x197d73)
    #10 jl_generate_fptr_impl ??:? (libjulia-codegen-debug.so.1.10+0x19789d)
    #11 jl_compile_method_internal ??:? (libjulia-internal-debug.so.1.10+0x5fba1)
    #12 gen_cfun_wrapper(llvm::Module*, jl_codegen_params_t&, function_sig_t const&, _jl_value_t*, char const*, _jl_value_t*, _jl_method_instance_t*, jl_unionall_t*, jl_svec_t*, jl_array_t**) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x499e7)
    #13 emit_cfunction(jl_codectx_t&, _jl_value_t*, jl_cgval_t const&, _jl_value_t*, jl_svec_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x8ed13)
    #14 emit_expr(jl_codectx_t&, _jl_value_t*, long) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x7bf44)
    #15 emit_ssaval_assign(jl_codectx_t&, long, _jl_value_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xf00f5)
    #16 emit_stmtpos(jl_codectx_t&, _jl_value_t*, int) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xeca03)
    #17 emit_function(llvm::orc::ThreadSafeModule&, _jl_method_instance_t*, _jl_code_info_t*, _jl_value_t*, jl_codegen_params_t&) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x58240)
    #18 jl_emit_code(llvm::orc::ThreadSafeModule&, _jl_method_instance_t*, _jl_code_info_t*, _jl_value_t*, jl_codegen_params_t&) :? (libjulia-codegen-debug.so.1.10+0x4ea72)
    #19 jl_emit_codeinst(llvm::orc::ThreadSafeModule&, _jl_code_instance_t*, _jl_code_info_t*, jl_codegen_params_t&) :? (libjulia-codegen-debug.so.1.10+0x5b0bf)
    #20 jl_compile_workqueue(jl_codegen_params_t&, llvm::Module&, CompilationPolicy) :? (libjulia-codegen-debug.so.1.10+0x5c5eb)
    #21 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long, llvm::orc::ThreadSafeContext, bool) jitlayers.cpp:? (libjulia-codegen-debug.so.1.10+0x197ff7)
    #22 jl_generate_fptr_impl ??:? (libjulia-codegen-debug.so.1.10+0x19789d)
    #23 jl_compile_method_internal ??:? (libjulia-internal-debug.so.1.10+0x5fba1)
    #24 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65a50)
    #25 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #26 jl_apply toplevel.c:? (libjulia-internal-debug.so.1.10+0xe845c)
    #27 jl_module_run_initializer :? (libjulia-internal-debug.so.1.10+0xe82ad)
    #28 _finish_julia_init init.c:? (libjulia-internal-debug.so.1.10+0xb0cac)
    #29 julia_init ??:? (libjulia-internal-debug.so.1.10+0xb0526)
    #30 jl_repl_entrypoint ??:? (libjulia-internal-debug.so.1.10+0x156772)
    #31 jl_load_repl ??:? (libjulia-debug.so.1.10+0x13693)
    #32 main ??:? (julia-debug+0xe8ddd)

    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message

  Mutex M0 acquired here while holding mutex M1 in main thread:
    #0 pthread_mutex_lock /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_common_interceptors.inc:4457 (julia-debug+0x6fac8)
    #1 __gthread_mutex_lock(pthread_mutex_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x60913)
    #2 __gthread_recursive_mutex_lock(pthread_mutex_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xf1b15)
    #3 std::recursive_mutex::lock() :? (libjulia-codegen-debug.so.1.10+0x18b535)
    #4 std::unique_lock<std::recursive_mutex>::lock() :? (libjulia-codegen-debug.so.1.10+0x18b4db)
    #5 std::unique_lock<std::recursive_mutex>::unique_lock(std::recursive_mutex&) :? (libjulia-codegen-debug.so.1.10+0x18b39e)
    #6 llvm::orc::ThreadSafeContext::Lock::Lock(std::shared_ptr<llvm::orc::ThreadSafeContext::State>) :? (libjulia-codegen-debug.so.1.10+0x18b0d5)
    #7 llvm::orc::ThreadSafeContext::getLock() const :? (libjulia-codegen-debug.so.1.10+0x18af26)
    #8 jl_codegen_params_t::jl_codegen_params_t(llvm::orc::ThreadSafeContext, llvm::DataLayout, llvm::Triple) :? (libjulia-codegen-debug.so.1.10+0x1b08d9)
    #9 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long, llvm::orc::ThreadSafeContext, bool) jitlayers.cpp:? (libjulia-codegen-debug.so.1.10+0x197d73)
    #10 jl_generate_fptr_impl ??:? (libjulia-codegen-debug.so.1.10+0x19789d)
    #11 jl_compile_method_internal ??:? (libjulia-internal-debug.so.1.10+0x5fba1)
    #12 gen_cfun_wrapper(llvm::Module*, jl_codegen_params_t&, function_sig_t const&, _jl_value_t*, char const*, _jl_value_t*, _jl_method_instance_t*, jl_unionall_t*, jl_svec_t*, jl_array_t**) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x499e7)
    #13 emit_cfunction(jl_codectx_t&, _jl_value_t*, jl_cgval_t const&, _jl_value_t*, jl_svec_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x8ed13)
    #14 emit_expr(jl_codectx_t&, _jl_value_t*, long) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x7bf44)
    #15 emit_ssaval_assign(jl_codectx_t&, long, _jl_value_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xf00f5)
    #16 emit_stmtpos(jl_codectx_t&, _jl_value_t*, int) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xeca03)
    #17 emit_function(llvm::orc::ThreadSafeModule&, _jl_method_instance_t*, _jl_code_info_t*, _jl_value_t*, jl_codegen_params_t&) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x58240)
    #18 jl_emit_code(llvm::orc::ThreadSafeModule&, _jl_method_instance_t*, _jl_code_info_t*, _jl_value_t*, jl_codegen_params_t&) :? (libjulia-codegen-debug.so.1.10+0x4ea72)
    #19 jl_emit_codeinst(llvm::orc::ThreadSafeModule&, _jl_code_instance_t*, _jl_code_info_t*, jl_codegen_params_t&) :? (libjulia-codegen-debug.so.1.10+0x5b0bf)
    #20 jl_compile_workqueue(jl_codegen_params_t&, llvm::Module&, CompilationPolicy) :? (libjulia-codegen-debug.so.1.10+0x5c5eb)
    #21 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long, llvm::orc::ThreadSafeContext, bool) jitlayers.cpp:? (libjulia-codegen-debug.so.1.10+0x197ff7)
    #22 jl_generate_fptr_impl ??:? (libjulia-codegen-debug.so.1.10+0x19789d)
    #23 jl_compile_method_internal ??:? (libjulia-internal-debug.so.1.10+0x5fba1)
    #24 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65a50)
    #25 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #26 jl_apply builtins.c:? (libjulia-internal-debug.so.1.10+0x8c0bc)
    #27 jl_f__call_in_world ??:? (libjulia-internal-debug.so.1.10+0x8c2f2)
    #28 <null> <null> (0x7f4cd6937493)
    #29 <null> <null> (0x7f4cc2f03e48)
    #30 <null> <null> (0x7f4cc29021de)
    #31 <null> <null> (0x7f4cc0d0adb7)
    #32 <null> <null> (0x7f4cc0b09962)
    #33 <null> <null> (0x7f4cc0b09a7d)
    #34 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65aef)
    #35 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #36 jl_apply jlapi.c:? (libjulia-internal-debug.so.1.10+0x15457c)
    #37 true_main jlapi.c:? (libjulia-internal-debug.so.1.10+0x156a28)
    #38 jl_repl_entrypoint ??:? (libjulia-internal-debug.so.1.10+0x1567df)
    #39 jl_load_repl ??:? (libjulia-debug.so.1.10+0x13693)
    #40 main ??:? (julia-debug+0xe8ddd)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) codegen.cpp:? in __gthread_mutex_lock(pthread_mutex_t*)
==================
└ Collect (Basic: ◓ , REPL 4/39: ◓ 737) => Execute ◓ 39ThreadSanitizer: CHECK failed: sanitizer_allocator_secondary.h:297 "((IsAligned(p, page_size_))) != (0)" (0x0, 0x0) (tid=2157108)
    #0 __tsan::CheckUnwind() /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_rtl.cpp:672 (julia-debug+0xc5875)
    #1 __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /workspace/srcdir/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_termination.cpp:86 (julia-debug+0x3dc0b)
    #2 __sanitizer::LargeMmapAllocator<__tsan::MapUnmapCallback, __sanitizer::LargeMmapAllocatorPtrArrayDynamic, __sanitizer::LocalAddressSpaceView>::GetHeader(unsigned long) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_allocator_secondary.h:297 (julia-debug+0xc2ffc)
    #3 __sanitizer::LargeMmapAllocator<__tsan::MapUnmapCallback, __sanitizer::LargeMmapAllocatorPtrArrayDynamic, __sanitizer::LocalAddressSpaceView>::GetHeader(void const*) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_allocator_secondary.h:301 (julia-debug+0xc2ffc)
    #4 __sanitizer::LargeMmapAllocator<__tsan::MapUnmapCallback, __sanitizer::LargeMmapAllocatorPtrArrayDynamic, __sanitizer::LocalAddressSpaceView>::Deallocate(__sanitizer::AllocatorStats*, void*) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_allocator_secondary.h:135 (julia-debug+0xc2ffc)
    #5 __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator64<__tsan::AP64>, __sanitizer::LargeMmapAllocatorPtrArrayDynamic>::Deallocate(__sanitizer::SizeClassAllocator64LocalCache<__sanitizer::SizeClassAllocator64<__tsan::AP64> >*, void*) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_allocator_combined.h:94 (julia-debug+0xc2ffc)
    #6 __tsan::user_free(__tsan::ThreadState*, unsigned long, void*, bool) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_mman.cpp:214 (julia-debug+0xc2ffc)
    #7 free /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:727 (julia-debug+0x65035)
    #8 jl_uv_closeHandle jl_uv.c:? (libjulia-internal-debug.so.1.10+0xf1ec9)
    #9 uv__finish_close core.c:? (libjulia-internal-debug.so.1.10+0x24c21f)
    #10 uv__run_closing_handles core.c:? (libjulia-internal-debug.so.1.10+0x24c265)
    #11 uv_run ??:? (libjulia-internal-debug.so.1.10+0x24c51f)
    #12 ijl_task_get_next ??:? (libjulia-internal-debug.so.1.10+0x122280)
    #13 <null> <null> (0x7fc62bb03249)
    #14 <null> <null> (0x7fc62b70617e)
    #15 <null> <null> (0x7fc62b701da4)
    #16 <null> <null> (0x7fc6304f5640)
    #17 <null> <null> (0x7fc6304e5993)
    #18 <null> <null> (0x7fc6304e5ad5)
    #19 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65aef)
    #20 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #21 jl_apply task.c:? (libjulia-internal-debug.so.1.10+0xb2a5c)
    #22 start_task task.c:? (libjulia-internal-debug.so.1.10+0xb6f15)

*** This error is usually fixed by running `make clean`. If the error persists, try `make cleanall`. ***
make[1]: *** [/home/liozou/julia-1.10-tsan/sysimage.mk:94: /home/liozou/julia-1.10-tsan/tsan-build/tsan/usr/lib/julia/sys-debug-o.a] Error 1
make[1]: Target 'sysimg-debug' not remade because of errors.
make: *** [/home/liozou/julia-1.10-tsan/Makefile:103: julia-sysimg-debug] Error 2
    CC tsan-build/tsan/usr/lib/libccalltest.so
    CC tsan-build/tsan/usr/lib/libllvmcalltest.so
make: Target 'default' not remade because of errors.
liozou@lavoisier:~/julia-1.10-tsan$ /usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x23
/usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x23
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=2166413)
  Cycle in lock order graph: M0 (0x7b1000025458) => M1 (0x7b10000343d8) => M0

  Mutex M1 acquired here while holding mutex M0 in main thread:
    #0 pthread_mutex_lock /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_common_interceptors.inc:4457 (julia-debug+0x6fac8)
    #1 __gthread_mutex_lock(pthread_mutex_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x60913)
    #2 __gthread_recursive_mutex_lock(pthread_mutex_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xf1b15)
    #3 std::recursive_mutex::lock() :? (libjulia-codegen-debug.so.1.10+0x18b535)
    #4 std::unique_lock<std::recursive_mutex>::lock() :? (libjulia-codegen-debug.so.1.10+0x18b4db)
    #5 std::unique_lock<std::recursive_mutex>::unique_lock(std::recursive_mutex&) :? (libjulia-codegen-debug.so.1.10+0x18b39e)
    #6 llvm::orc::ThreadSafeContext::Lock::Lock(std::shared_ptr<llvm::orc::ThreadSafeContext::State>) :? (libjulia-codegen-debug.so.1.10+0x18b0d5)
    #7 llvm::orc::ThreadSafeContext::getLock() const :? (libjulia-codegen-debug.so.1.10+0x18af26)
    #8 jl_codegen_params_t::jl_codegen_params_t(llvm::orc::ThreadSafeContext, llvm::DataLayout, llvm::Triple) :? (libjulia-codegen-debug.so.1.10+0x1b08d9)
    #9 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long, llvm::orc::ThreadSafeContext, bool) jitlayers.cpp:? (libjulia-codegen-debug.so.1.10+0x197d73)
    #10 jl_generate_fptr_impl ??:? (libjulia-codegen-debug.so.1.10+0x19789d)
    #11 jl_compile_method_internal ??:? (libjulia-internal-debug.so.1.10+0x5fba1)
    #12 gen_cfun_wrapper(llvm::Module*, jl_codegen_params_t&, function_sig_t const&, _jl_value_t*, char const*, _jl_value_t*, _jl_method_instance_t*, jl_unionall_t*, jl_svec_t*, jl_array_t**) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x499e7)
    #13 emit_cfunction(jl_codectx_t&, _jl_value_t*, jl_cgval_t const&, _jl_value_t*, jl_svec_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x8ed13)
    #14 emit_expr(jl_codectx_t&, _jl_value_t*, long) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x7bf44)
    #15 emit_ssaval_assign(jl_codectx_t&, long, _jl_value_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xf00f5)
    #16 emit_stmtpos(jl_codectx_t&, _jl_value_t*, int) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xeca03)
    #17 emit_function(llvm::orc::ThreadSafeModule&, _jl_method_instance_t*, _jl_code_info_t*, _jl_value_t*, jl_codegen_params_t&) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x58240)
    #18 jl_emit_code(llvm::orc::ThreadSafeModule&, _jl_method_instance_t*, _jl_code_info_t*, _jl_value_t*, jl_codegen_params_t&) :? (libjulia-codegen-debug.so.1.10+0x4ea72)
    #19 jl_emit_codeinst(llvm::orc::ThreadSafeModule&, _jl_code_instance_t*, _jl_code_info_t*, jl_codegen_params_t&) :? (libjulia-codegen-debug.so.1.10+0x5b0bf)
    #20 jl_compile_workqueue(jl_codegen_params_t&, llvm::Module&, CompilationPolicy) :? (libjulia-codegen-debug.so.1.10+0x5c5eb)
    #21 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long, llvm::orc::ThreadSafeContext, bool) jitlayers.cpp:? (libjulia-codegen-debug.so.1.10+0x197ff7)
    #22 jl_generate_fptr_impl ??:? (libjulia-codegen-debug.so.1.10+0x19789d)
    #23 jl_compile_method_internal ??:? (libjulia-internal-debug.so.1.10+0x5fba1)
    #24 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65a50)
    #25 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #26 jl_apply toplevel.c:? (libjulia-internal-debug.so.1.10+0xe845c)
    #27 jl_module_run_initializer :? (libjulia-internal-debug.so.1.10+0xe82ad)
    #28 _finish_julia_init init.c:? (libjulia-internal-debug.so.1.10+0xb0cac)
    #29 julia_init ??:? (libjulia-internal-debug.so.1.10+0xb0526)
    #30 jl_repl_entrypoint ??:? (libjulia-internal-debug.so.1.10+0x156772)
    #31 jl_load_repl ??:? (libjulia-debug.so.1.10+0x13693)
    #32 main ??:? (julia-debug+0xe8ddd)

    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message

  Mutex M0 acquired here while holding mutex M1 in main thread:
    #0 pthread_mutex_lock /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_common_interceptors.inc:4457 (julia-debug+0x6fac8)
    #1 __gthread_mutex_lock(pthread_mutex_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x60913)
    #2 __gthread_recursive_mutex_lock(pthread_mutex_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xf1b15)
    #3 std::recursive_mutex::lock() :? (libjulia-codegen-debug.so.1.10+0x18b535)
    #4 std::unique_lock<std::recursive_mutex>::lock() :? (libjulia-codegen-debug.so.1.10+0x18b4db)
    #5 std::unique_lock<std::recursive_mutex>::unique_lock(std::recursive_mutex&) :? (libjulia-codegen-debug.so.1.10+0x18b39e)
    #6 llvm::orc::ThreadSafeContext::Lock::Lock(std::shared_ptr<llvm::orc::ThreadSafeContext::State>) :? (libjulia-codegen-debug.so.1.10+0x18b0d5)
    #7 llvm::orc::ThreadSafeContext::getLock() const :? (libjulia-codegen-debug.so.1.10+0x18af26)
    #8 jl_codegen_params_t::jl_codegen_params_t(llvm::orc::ThreadSafeContext, llvm::DataLayout, llvm::Triple) :? (libjulia-codegen-debug.so.1.10+0x1b08d9)
    #9 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long, llvm::orc::ThreadSafeContext, bool) jitlayers.cpp:? (libjulia-codegen-debug.so.1.10+0x197d73)
    #10 jl_generate_fptr_impl ??:? (libjulia-codegen-debug.so.1.10+0x19789d)
    #11 jl_compile_method_internal ??:? (libjulia-internal-debug.so.1.10+0x5fba1)
    #12 gen_cfun_wrapper(llvm::Module*, jl_codegen_params_t&, function_sig_t const&, _jl_value_t*, char const*, _jl_value_t*, _jl_method_instance_t*, jl_unionall_t*, jl_svec_t*, jl_array_t**) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x499e7)
    #13 emit_cfunction(jl_codectx_t&, _jl_value_t*, jl_cgval_t const&, _jl_value_t*, jl_svec_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x8ed13)
    #14 emit_expr(jl_codectx_t&, _jl_value_t*, long) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x7bf44)
    #15 emit_ssaval_assign(jl_codectx_t&, long, _jl_value_t*) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xf00f5)
    #16 emit_stmtpos(jl_codectx_t&, _jl_value_t*, int) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0xeca03)
    #17 emit_function(llvm::orc::ThreadSafeModule&, _jl_method_instance_t*, _jl_code_info_t*, _jl_value_t*, jl_codegen_params_t&) codegen.cpp:? (libjulia-codegen-debug.so.1.10+0x58240)
    #18 jl_emit_code(llvm::orc::ThreadSafeModule&, _jl_method_instance_t*, _jl_code_info_t*, _jl_value_t*, jl_codegen_params_t&) :? (libjulia-codegen-debug.so.1.10+0x4ea72)
    #19 jl_emit_codeinst(llvm::orc::ThreadSafeModule&, _jl_code_instance_t*, _jl_code_info_t*, jl_codegen_params_t&) :? (libjulia-codegen-debug.so.1.10+0x5b0bf)
    #20 jl_compile_workqueue(jl_codegen_params_t&, llvm::Module&, CompilationPolicy) :? (libjulia-codegen-debug.so.1.10+0x5c5eb)
    #21 _jl_compile_codeinst(_jl_code_instance_t*, _jl_code_info_t*, unsigned long, llvm::orc::ThreadSafeContext, bool) jitlayers.cpp:? (libjulia-codegen-debug.so.1.10+0x197ff7)
    #22 jl_generate_fptr_impl ??:? (libjulia-codegen-debug.so.1.10+0x19789d)
    #23 jl_compile_method_internal ??:? (libjulia-internal-debug.so.1.10+0x5fba1)
    #24 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65a50)
    #25 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #26 jl_apply builtins.c:? (libjulia-internal-debug.so.1.10+0x8c0bc)
    #27 jl_f__call_in_world ??:? (libjulia-internal-debug.so.1.10+0x8c2f2)
    #28 jl_fptr_args :? (libjulia-internal-debug.so.1.10+0x52046)
    #29 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65985)
    #30 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #31 jl_apply builtins.c:? (libjulia-internal-debug.so.1.10+0x8c0bc)
    #32 do_apply builtins.c:? (libjulia-internal-debug.so.1.10+0x8bb73)
    #33 jl_f__apply_iterate ??:? (libjulia-internal-debug.so.1.10+0x8a2e1)
    #34 <null> <null> (0x7f0a668032c9)
    #35 jl_fptr_args :? (libjulia-internal-debug.so.1.10+0x52046)
    #36 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65aef)
    #37 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #38 jl_apply builtins.c:? (libjulia-internal-debug.so.1.10+0x8c0bc)
    #39 do_apply builtins.c:? (libjulia-internal-debug.so.1.10+0x8bb73)
    #40 jl_f__apply_iterate ??:? (libjulia-internal-debug.so.1.10+0x8a2e1)
    #41 <null> <null> (0x7f0a66808322)
    #42 jl_fptr_args :? (libjulia-internal-debug.so.1.10+0x52046)
    #43 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65aef)
    #44 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #45 jl_apply builtins.c:? (libjulia-internal-debug.so.1.10+0x8c0bc)
    #46 do_apply builtins.c:? (libjulia-internal-debug.so.1.10+0x8bb73)
    #47 jl_f__apply_iterate ??:? (libjulia-internal-debug.so.1.10+0x8a2e1)
    #48 <null> <null> (0x7f0a6680d345)
    #49 jl_fptr_args :? (libjulia-internal-debug.so.1.10+0x52046)
    #50 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65aef)
    #51 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #52 jl_apply builtins.c:? (libjulia-internal-debug.so.1.10+0x8c0bc)
    #53 do_apply builtins.c:? (libjulia-internal-debug.so.1.10+0x8bb73)
    #54 jl_f__apply_iterate ??:? (libjulia-internal-debug.so.1.10+0x8a2e1)
    #55 <null> <null> (0x7f0a66a033e2)
    #56 jl_fptr_args :? (libjulia-internal-debug.so.1.10+0x52046)
    #57 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65aef)
    #58 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #59 <null> <null> (0x7f0a69e0235d)
    #60 <null> <null> (0x7f0a68608cfa)
    #61 <null> <null> (0x7f0a74fd128e)
    #62 <null> <null> (0x7f0a74fa5db7)
    #63 <null> <null> (0x7f0a74fa5f2d)
    #64 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65aef)
    #65 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #66 jl_apply jlapi.c:? (libjulia-internal-debug.so.1.10+0x15457c)
    #67 true_main jlapi.c:? (libjulia-internal-debug.so.1.10+0x156a28)
    #68 jl_repl_entrypoint ??:? (libjulia-internal-debug.so.1.10+0x1567df)
    #69 jl_load_repl ??:? (libjulia-debug.so.1.10+0x13693)
    #70 main ??:? (julia-debug+0xe8ddd)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) codegen.cpp:? in __gthread_mutex_lock(pthread_mutex_t*)
==================

[2166413] signal (11.2): Segmentation fault
in expression starting at none:0

After the segfault TSAN blurts a non-stopping flow of warnings about signal-unsafe call inside of a signal that look almost all identical like:

WARNING: ThreadSanitizer: signal-unsafe call inside of a signal (pid=2152551)
    #0 malloc /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:667 (julia-debug+0x8390d)
    #1 operator new(unsigned long) /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/new_op.cc:50 (libstdc++.so.6+0xb191b)
    #2 jl_getDylibFunctionInfo(jl_frame_t**, unsigned long, int, int) debuginfo.cpp:? (libjulia-codegen-debug.so.1.10+0x25acad)
    #3 jl_getFunctionInfo_impl ??:? (libjulia-codegen-debug.so.1.10+0x259db5)
    #4 jl_print_native_codeloc :? (libjulia-internal-debug.so.1.10+0x124c5e)
    #5 jl_print_bt_entry_codeloc :? (libjulia-internal-debug.so.1.10+0x124e1a)
    #6 jl_critical_error :? (libjulia-internal-debug.so.1.10+0x15a9e1)
    #7 sigdie_handler signal-handling.c:? (libjulia-internal-debug.so.1.10+0x159f6d)
    #8 segv_handler signal-handling.c:? (libjulia-internal-debug.so.1.10+0x15bd4a)
    #9 __tsan::CallUserSignalHandler(__tsan::ThreadState*, bool, bool, int, __sanitizer::__sanitizer_siginfo*, void*) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2025 (julia-debug+0x63929)
    #10 <null> <null> (0x7fb80c2087f8)
    #11 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65aef)
    #12 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #13 run_finalizer gc.c:? (libjulia-internal-debug.so.1.10+0x136d53)
    #14 jl_gc_run_finalizers_in_list gc.c:? (libjulia-internal-debug.so.1.10+0x128592)
    #15 run_finalizers gc.c:? (libjulia-internal-debug.so.1.10+0x12718d)
    #16 jl_gc_run_all_finalizers :? (libjulia-internal-debug.so.1.10+0x1276d2)
    #17 ijl_atexit_hook ??:? (libjulia-internal-debug.so.1.10+0xaf6cb)
    #18 jl_repl_entrypoint ??:? (libjulia-internal-debug.so.1.10+0x1567ea)
    #19 jl_load_repl ??:? (libjulia-debug.so.1.10+0x13693)
    #20 main ??:? (julia-debug+0xe8ddd)

SUMMARY: ThreadSanitizer: signal-unsafe call inside of a signal /workspace/srcdir/gcc-12.1.0/libstdc++-v3/libsupc++/new_op.cc:50 in operator new(unsigned long)

with only lines 0 and 1 changing, like

WARNING: ThreadSanitizer: signal-unsafe call inside of a signal (pid=2166413)
    #0 free /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:726 (julia-debug+0x65030)
    #1 void std::vector<llvm::DWARFAbbreviationDeclaration, std::allocator<llvm::DWARFAbbreviationDeclaration> >::_M_realloc_insert<llvm::DWARFAbbreviationDeclaration>(__gnu_cxx::__normal_iterator<llvm::DWARFAbbreviationDeclaration*, std::vector<llvm::DWARFAbbreviationDeclaration, std::allocator<llvm::DWARFAbbreviationDeclaration> > >, llvm::DWARFAbbreviationDeclaration&&) ??:? (libLLVM-15jl.so+0x2a47024)
   [#2 to #20 is like the previous one]

SUMMARY: ThreadSanitizer: signal-unsafe call inside of a signal ??:? in void std::vector<llvm::DWARFAbbreviationDeclaration, std::allocator<llvm::DWARFAbbreviationDeclaration> >::_M_realloc_insert<llvm::DWARFAbbreviationDeclaration>(__gnu_cxx::__normal_iterator<llvm::DWARFAbbreviationDeclaration*, std::vector<llvm::DWARFAbbreviationDeclaration, std::allocator<llvm::DWARFAbbreviationDeclaration> > >, llvm::DWARFAbbreviationDeclaration&&)

Any idea about what to do is greatly appreciated!

@maleadt
Copy link
Member

maleadt commented Oct 20, 2023

You could temporarily disable the backtrace printing in jl_critical_error. In any case, TSAN complaining about us doing a malloc from within sigdie_handler seems like a reasonable thing, as this is a frequent cause of hangs (e.g. as recently encountered by @jpsamaroo with AMDGPU.jl, where memory corruption caused malloc to abort, at which point the heap lock was taken and allocating during the signal handler caused a deadlock).

@Liozou
Copy link
Member Author

Liozou commented Oct 20, 2023

Thanks for the tip, that does remove all those (not-quite-so-)spurious warnings (which may warrant a separate issue, considering what you report?).
But underneath it all, TSAN still does not compile: if I extract the error lines that do not belong to a WARNING: ThreadSanitizer: ... block, I have the following:

Reduced error log
/usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x23
/usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x23

/usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x23
/usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x23

ThreadSanitizer: CHECK failed: sanitizer_allocator_secondary.h:297 "((IsAligned(p, page_size_))) != (0)" (0x0, 0x0) (tid=2307538)
    #0 __tsan::CheckUnwind() /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_rtl.cpp:672 (julia-debug+0xc5875)
    #1 __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /workspace/srcdir/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_termination.cpp:86 (julia-debug+0x3dc0b)
    #2 __sanitizer::LargeMmapAllocator<__tsan::MapUnmapCallback, __sanitizer::LargeMmapAllocatorPtrArrayDynamic, __sanitizer::LocalAddressSpaceView>::GetHeader(unsigned long) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_allocator_secondary.h:297 (julia-debug+0xc2ffc)
    #3 __sanitizer::LargeMmapAllocator<__tsan::MapUnmapCallback, __sanitizer::LargeMmapAllocatorPtrArrayDynamic, __sanitizer::LocalAddressSpaceView>::GetHeader(void const*) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_allocator_secondary.h:301 (julia-debug+0xc2ffc)
    #4 __sanitizer::LargeMmapAllocator<__tsan::MapUnmapCallback, __sanitizer::LargeMmapAllocatorPtrArrayDynamic, __sanitizer::LocalAddressSpaceView>::Deallocate(__sanitizer::AllocatorStats*, void*) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_allocator_secondary.h:135 (julia-debug+0xc2ffc)
    #5 __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator64<__tsan::AP64>, __sanitizer::LargeMmapAllocatorPtrArrayDynamic>::Deallocate(__sanitizer::SizeClassAllocator64LocalCache<__sanitizer::SizeClassAllocator64<__tsan::AP64> >*, void*) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_allocator_combined.h:94 (julia-debug+0xc2ffc)
    #6 __tsan::user_free(__tsan::ThreadState*, unsigned long, void*, bool) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_mman.cpp:214 (julia-debug+0xc2ffc)
    #7 free /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:727 (julia-debug+0x65035)
    #8 jl_uv_closeHandle jl_uv.c:? (libjulia-internal-debug.so.1.10+0xf1ec9)
    #9 uv__finish_close core.c:? (libjulia-internal-debug.so.1.10+0x24c12f)
    #10 uv__run_closing_handles core.c:? (libjulia-internal-debug.so.1.10+0x24c175)
    #11 uv_run ??:? (libjulia-internal-debug.so.1.10+0x24c42f)
    #12 ijl_task_get_next ??:? (libjulia-internal-debug.so.1.10+0x122280)
    #13 <null> <null> (0x7f76a290b249)
    #14 <null> <null> (0x7f76a250d17e)
    #15 <null> <null> (0x7f76a2508da4)
    #16 <null> <null> (0x7f7695905640)
    #17 <null> <null> (0x7f7695702993)
    #18 <null> <null> (0x7f7695702ad5)
    #19 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65aef)
    #20 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #21 jl_apply task.c:? (libjulia-internal-debug.so.1.10+0xb2a5c)
    #22 start_task task.c:? (libjulia-internal-debug.so.1.10+0xb6f15)

/usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x23
make[1]: *** [/home/liozou/julia-1.10-tsan/sysimage.mk:94: /home/liozou/julia-1.10-tsan/tsan-build/tsan/usr/lib/julia/sys-debug-o.a] Error 1
make[1]: Target 'sysimg-debug' not remade because of errors.
make: *** [/home/liozou/julia-1.10-tsan/Makefile:103: julia-sysimg-debug] Error 2

/usr/bin/addr2line: DWARF error: invalid or unhandled FORM value: 0x23

make: Target 'default' not remade because of errors.

[2308037] signal (11.2): Segmentation fault

ThreadSanitizer: CHECK failed: sanitizer_allocator_secondary.h:297 "((IsAligned(p, page_size_))) != (0)" (0x0, 0x0) (tid=2307891)
    #0 __tsan::CheckUnwind() /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_rtl.cpp:672 (julia-debug+0xc5875)
    #1 __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /workspace/srcdir/llvm-project/compiler-rt/lib/sanitizer_common/sanitizer_termination.cpp:86 (julia-debug+0x3dc0b)
    #2 __sanitizer::LargeMmapAllocator<__tsan::MapUnmapCallback, __sanitizer::LargeMmapAllocatorPtrArrayDynamic, __sanitizer::LocalAddressSpaceView>::GetHeader(unsigned long) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_allocator_secondary.h:297 (julia-debug+0xc2ffc)
    #3 __sanitizer::LargeMmapAllocator<__tsan::MapUnmapCallback, __sanitizer::LargeMmapAllocatorPtrArrayDynamic, __sanitizer::LocalAddressSpaceView>::GetHeader(void const*) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_allocator_secondary.h:301 (julia-debug+0xc2ffc)
    #4 __sanitizer::LargeMmapAllocator<__tsan::MapUnmapCallback, __sanitizer::LargeMmapAllocatorPtrArrayDynamic, __sanitizer::LocalAddressSpaceView>::Deallocate(__sanitizer::AllocatorStats*, void*) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_allocator_secondary.h:135 (julia-debug+0xc2ffc)
    #5 __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator64<__tsan::AP64>, __sanitizer::LargeMmapAllocatorPtrArrayDynamic>::Deallocate(__sanitizer::SizeClassAllocator64LocalCache<__sanitizer::SizeClassAllocator64<__tsan::AP64> >*, void*) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/../../sanitizer_common/sanitizer_allocator_combined.h:94 (julia-debug+0xc2ffc)
    #6 __tsan::user_free(__tsan::ThreadState*, unsigned long, void*, bool) /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_mman.cpp:214 (julia-debug+0xc2ffc)
    #7 free /workspace/srcdir/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:727 (julia-debug+0x65035)
    #8 jl_uv_closeHandle jl_uv.c:? (libjulia-internal-debug.so.1.10+0xf1ec9)
    #9 uv__finish_close core.c:? (libjulia-internal-debug.so.1.10+0x24c12f)
    #10 uv__run_closing_handles core.c:? (libjulia-internal-debug.so.1.10+0x24c175)
    #11 uv_run ??:? (libjulia-internal-debug.so.1.10+0x24c42f)
    #12 ijl_task_get_next ??:? (libjulia-internal-debug.so.1.10+0x122280)
    #13 <null> <null> (0x7f88a9f241b0)
    #14 <null> <null> (0x7f88a9f0d17e)
    #15 <null> <null> (0x7f88a9de4c68)
    #16 <null> <null> (0x7f888f32156b)
    #17 <null> <null> (0x7f8875d0d114)
    #18 <null> <null> (0x7f8875d0d218)
    #19 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65aef)
    #20 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #21 <null> <null> (0x7f888efb97fa)
    #22 <null> <null> (0x7f887b9051a7)
    #23 <null> <null> (0x7f887b90528c)
    #24 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65aef)
    #25 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #26 jl_apply interpreter.c:? (libjulia-internal-debug.so.1.10+0xaae5c)
    #27 do_call interpreter.c:? (libjulia-internal-debug.so.1.10+0xaa9d9)
    #28 eval_value interpreter.c:? (libjulia-internal-debug.so.1.10+0xa7b73)
    #29 eval_stmt_value interpreter.c:? (libjulia-internal-debug.so.1.10+0xa9c45)
    #30 eval_body interpreter.c:? (libjulia-internal-debug.so.1.10+0xa5d3e)
    #31 jl_interpret_toplevel_thunk :? (libjulia-internal-debug.so.1.10+0xa6e0d)
    #32 jl_toplevel_eval_flex :? (libjulia-internal-debug.so.1.10+0xec1b4)
    #33 jl_toplevel_eval_flex :? (libjulia-internal-debug.so.1.10+0xeb868)
    #34 ijl_toplevel_eval ??:? (libjulia-internal-debug.so.1.10+0xeebbc)
    #35 ijl_toplevel_eval_in ??:? (libjulia-internal-debug.so.1.10+0xef01c)
    #36 <null> <null> (0x7f88873089f8)
    #37 <null> <null> (0x7f888730afa3)
    #38 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65985)
    #39 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #40 <null> <null> (0x7f8896305fb4)
    #41 <null> <null> (0x7f889470adb7)
    #42 <null> <null> (0x7f889450b962)
    #43 <null> <null> (0x7f889450ba7d)
    #44 _jl_invoke gf.c:? (libjulia-internal-debug.so.1.10+0x65aef)
    #45 ijl_apply_generic ??:? (libjulia-internal-debug.so.1.10+0x65bd9)
    #46 jl_apply jlapi.c:? (libjulia-internal-debug.so.1.10+0x15457c)
    #47 true_main jlapi.c:? (libjulia-internal-debug.so.1.10+0x156a28)
    #48 jl_repl_entrypoint ??:? (libjulia-internal-debug.so.1.10+0x1567df)
    #49 jl_load_repl ??:? (libjulia-debug.so.1.10+0x13693)

The DWARF issue looks like something got stuck with a version of DWARF prior to v5 (where DW_FORM_rnglistx was introduced as 0x23) while something else expects DWARF v5. I don't know if that's expected... And I don't know what to make of the failed TSAN checks.
What's also surprising to me is that the build command apparently spawns a TSAN subprocess that outlives the segfaulted make (and which was responsible for all the previous ThreadSanitizer: signal-unsafe call inside of a signal warnings long after the make command had stopped)

@brenhinkeller brenhinkeller added the domain:building Build system, or building Julia or its dependencies label Oct 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain:building Build system, or building Julia or its dependencies
Projects
None yet
Development

No branches or pull requests

4 participants