thread_local destructors not called at thread exit

[jeremyd2019]

I stepped through the disassembly, and the tls value that mingw was storing its dtor list in was NULL at the time the tls_callback was called. It appears that emutls is cleaned up before mingw's tls_callback is called for DLL_THREAD_DETACH (3).

Verified on both i686 and x86_64.

/cc @mstorsjo author of mingw's __cxa_thread_atexit

Thread 5 hit Breakpoint 2, 0x00406d70 in emutls_destroy ()
(gdb) c
Continuing.

Thread 5 hit Breakpoint 1, tls_callback (hDllHandle=0x400000, dwReason=3,
    lpReserved=0x0)
    at C:/_/M/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/tls_atexit.c:89
89      in C:/_/M/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/tls_atexit.c

Originally posted by @jeremyd2019 in #7071 (comment)

[jeremyd2019]

Test program:

#include <thread>
#include <string>

using namespace std;

thread_local string s("Hello");

int main()
{
        thread([] {
                s;
        }).join();
        return 0;
}

If built with g++ -std=c++11 -static-libgcc, breakpoints can be set on tls_callback and emutls_destroy (and std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() to see that it is never called).

[jeremyd2019]

Opened upstream bug https://sourceforge.net/p/mingw-w64/bugs/859/ once I found my old sourceforge account and got the password reset

[jeremyd2019]

I've been playing with this a little on and off. I pasted a patch to the mingw-w64 bug that allows their implementation to work with gcc, but I was concerned because the destructors would be run after emutls was destroyed, so all thread_locals would be NULL. Today I tried applying @lhmouse's patch from #7071 (comment) and overriding gcc/libstdc++'s configure test so it would use its implementation instead of the crt's. This also seems to work, probably at the proper time of thread tear-down.

[mstorsjo]

I've been playing with this a little on and off. I pasted a patch to the mingw-w64 bug that allows their implementation to work with gcc, but I was concerned because the destructors would be run after emutls was destroyed, so all thread_locals would be NULL. Today I tried applying @lhmouse's patch from #7071 (comment) and overriding gcc/libstdc++'s configure test so it would use its implementation instead of the crt's. This also seems to work, probably at the proper time of thread tear-down.

(commenting here also, for visibility, in addition to the mingw-w64 bugtracker); Your patch for mingw-w64 tls_atexit.c looks good and would be a good thing to have in any case. But you're right that it might be a bit problematic still when used together with emutls, as the final destructors are run after emutls has shut down. A GCC/libstdc++ option of forcing it to provide its own __cxa_thread_atexit could be a good thing to have.

Regarding correctness of when destructors are executed, IIRC the GCC/libstdc++ implementation doesn't run destructors for TLS objects owned by a particular DLL when a DLL is unloaded (but running those destructors later, which can be fatal, iirc I've seen a bug report for this somewhere as well). I've got a testcase that should, hopefully, test all the various combinations of threads and DLL loading/unloading, at https://github.com/mstorsjo/llvm-mingw/blob/master/test/tlstest-lib.cpp and https://github.com/mstorsjo/llvm-mingw/blob/master/test/tlstest-main.cpp.

[jeremyd2019]

Regarding correctness of when destructors are executed, IIRC the GCC/libstdc++ implementation doesn't run destructors for TLS objects owned by a particular DLL when a DLL is unloaded (but running those destructors later, which can be fatal, iirc I've seen a bug report for this somewhere as well).

I guess that explains https://github.com/gcc-mirror/gcc/blob/master/libstdc%2B%2B-v3/libsupc%2B%2B/atexit_thread.cc#L153-L158

I've got a testcase that should, hopefully, test all the various combinations of threads and DLL loading/unloading, at https://github.com/mstorsjo/llvm-mingw/blob/master/test/tlstest-lib.cpp and https://github.com/mstorsjo/llvm-mingw/blob/master/test/tlstest-main.cpp.

Log with current state:

main global ctor on thread 8328
main
main, starting thread1
threadfunc thread 3872
main local tls ctor on thread 3872
main, thread1 started
LoadLibrary tlstest-lib.dll
lib global ctor on thread 8328
LoadLibrary tlstest-lib.dll ret 6b740000
main, got func address, calling it
func
lib local tls ctor on thread 8328
lib global tls ctor on thread 8328
func end, thread 8328
main, starting thread2
threadfunc thread 1292
main local tls ctor on thread 1292
main, thread2 started
thread 3872 calling func
func
lib local tls ctor on thread 3872
lib global tls ctor on thread 3872
func end, thread 3872
main, thread1 work done
thread 1292 calling func
func
lib local tls ctor on thread 1292
lib global tls ctor on thread 1292
func end, thread 1292
main, thread2 work done
thread 3872 finishing
lib global tls dtor from thread 3872, now on 3872
lib local tls dtor from thread 3872, now on 3872
main, thread1 joined
FreeLibrary
lib global tls dtor from thread 8328, now on 8328
lib local tls dtor from thread 8328, now on 8328
lib_atexit
lib global dtor from thread 8328, now on 8328
FreeLibrary done
thread 1292 finishing
main, thread2 joined
main local tls ctor on thread 8328
main done
main atexit_func
main atexit_func
main global dtor from thread 8328, now on 8328
main local tls dtor from thread 8328, now on 8328

Log with gcc/libstdc++'s implementation:

main global ctor on thread 1188
main
main, starting thread1
threadfunc thread 7772
main local tls ctor on thread 7772
main, thread1 started
LoadLibrary tlstest-lib.dll
lib global ctor on thread 1188
LoadLibrary tlstest-lib.dll ret 6b740000
main, got func address, calling it
func
lib local tls ctor on thread 1188
lib global tls ctor on thread 1188
func end, thread 1188
main, starting thread2
threadfunc thread 1392
main local tls ctor on thread 1392
main, thread2 started
thread 7772 calling func
func
lib local tls ctor on thread 7772
lib global tls ctor on thread 7772
func end, thread 7772
main, thread1 work done
thread 1392 calling func
func
lib local tls ctor on thread 1392
lib global tls ctor on thread 1392
func end, thread 1392
main, thread2 work done
thread 7772 finishing
lib global tls dtor from thread 7772, now on 7772
lib local tls dtor from thread 7772, now on 7772
main local tls dtor from thread 7772, now on 7772
main, thread1 joined
FreeLibrary
FreeLibrary done
thread 1392 finishing
lib global tls dtor from thread 1392, now on 1392
lib local tls dtor from thread 1392, now on 1392
main local tls dtor from thread 1392, now on 1392
main, thread2 joined
main local tls ctor on thread 1188
main done
main atexit_func
main atexit_func
main global dtor from thread 1188, now on 1188
lib_atexit
lib global dtor from thread 1188, now on 1188
main local tls dtor from thread 1188, now on 1188
lib global tls dtor from thread 1188, now on 1188
lib local tls dtor from thread 1188, now on 1188

Log with patched mingw-w64 implementation

main global ctor on thread 236
main
main, starting thread1
threadfunc thread 4608
main local tls ctor on thread 4608
main, thread1 started
LoadLibrary tlstest-lib.dll
lib global ctor on thread 236
LoadLibrary tlstest-lib.dll ret 6b740000
main, got func address, calling it
func
lib local tls ctor on thread 236
lib global tls ctor on thread 236
func end, thread 236
main, starting thread2
threadfunc thread 356
main local tls ctor on thread 356
main, thread2 started
thread 4608 calling func
func
lib local tls ctor on thread 4608
lib global tls ctor on thread 4608
func end, thread 4608
main, thread1 work done
thread 356 calling func
func
lib local tls ctor on thread 356
lib global tls ctor on thread 356
func end, thread 356
main, thread2 work done
thread 4608 finishing
lib global tls dtor from thread 4608, now on 4608
lib local tls dtor from thread 4608, now on 4608
ð�â dtor from thread 4608, now on 4608
main, thread1 joined
FreeLibrary
lib global tls dtor from thread 236, now on 236
lib local tls dtor from thread 236, now on 236
lib_atexit
lib global dtor from thread 236, now on 236
FreeLibrary done
thread 356 finishing
main local tls dtor from thread 356, now on 356
main, thread2 joined
main local tls ctor on thread 236
main done
main atexit_func
main atexit_func
main global dtor from thread 236, now on 236
main local tls dtor from thread 236, now on 236

Log with VC2019:

main global ctor on thread 3068
main
main, starting thread1
threadfunc thread 10164
main local tls ctor on thread 10164
main, thread1 started
LoadLibrary tlstest-lib.dll
lib global ctor on thread 3068
lib global tls ctor on thread 3068
LoadLibrary tlstest-lib.dll ret 60680000
main, got func address, calling it
func
lib local tls ctor on thread 3068
func end, thread 3068
main, starting thread2
lib global tls ctor on thread 1004
threadfunc thread 1004
main local tls ctor on thread 1004
main, thread2 started
thread 10164 calling func
func
lib local tls ctor on thread 10164
lib global tls ctor on thread 10164
func end, thread 10164
main, thread1 work done
thread 1004 calling func
func
lib local tls ctor on thread 1004
func end, thread 1004
main, thread2 work done
thread 10164 finishing
lib global tls dtor from thread 10164, now on 10164
lib local tls dtor from thread 10164, now on 10164
main local tls dtor from thread 10164, now on 10164
main, thread1 joined
FreeLibrary
lib local tls dtor from thread 3068, now on 3068
lib global tls dtor from thread 3068, now on 3068
lib_atexit
lib global dtor from thread 3068, now on 3068
FreeLibrary done
thread 1004 finishing
main local tls dtor from thread 1004, now on 1004
main, thread2 joined
main local tls ctor on thread 3068
main done
main local tls dtor from thread 3068, now on 3068
main atexit_func
main atexit_func
main global dtor from thread 3068, now on 3068

[lhmouse]

If any static destructor is invoked prior to a TLS destructor then it would be another bug [1]. This might require mingw-w64 to override exit(). I will propose a patch later for this.

[1] https://en.cppreference.com/w/cpp/utility/program/exit: a) The last destructor for thread-local objects is sequenced-before the first destructor for a static object

[jeremyd2019]

I added logs to my prior comment for VC2019 and the current unpatched state of mingw-w64.