Unloaded dll compiled with exception support causes exe to crash on exit

[justanotheranonymoususer]

I'm not sure whether the issue is caused by one of the patches or whether it's a GCC bug. There are patches about std::mutex, mcrtdll, threads, all potentially related.

Reproduction

Create and compile dll.cc as following:

#include <mutex>
std::mutex x;

g++.exe dll.cc -shared -o test.dll

Create and compile exe.cc as following:

#include <windows.h>
int main()
{
    auto x = LoadLibrary("test.dll");
    MessageBox(0,0,0,0);
    FreeLibrary(x);
    MessageBox(0,0,0,0);
    return 0;
}

g++.exe exe.cc -o test.exe

Run test.exe and observe a crash after seeing both message boxes.
The crash happens in msvcrt.doexit, which tries to call a cleanup function inside the dll which was already freed.

[justanotheranonymoususer]

I debugged it a bit more, and I believe that I found the reason for the issue.

First, note that it's related to exceptions. Even a dll as simple as the following triggers the crash:

#include <vector>
void x()
{
    std::vector<int> v;
    v.push_back(2);
}

As for the root of the problem, it's caused by an _onexit function call here:
https://github.com/jmeubank/tdm-gcc-src/blob/6a41b3e46f25b072e82520fc21a1fe2ece8d1948/libgcc/config/i386/eh_shmem3_mingw.c#L203

The exported _onexit function in the dynamically linked msvcrt.dll library is called, which causes the callback to be registered process-wide. When the dll is unloaded, the function pointer stays there, and eventually causes the process to crash on unload.

Perhaps you can call the CRT's atexit function to fix this.

There's the __dllonexit function that is used in the runtime library, but I'm not sure you can use it to solve the issue.

[jmeubank]

Hi @justanotheranonymoususer ,

Thanks for spending the time to look into this issue!

Microsoft's docs state:

In the case when _onexit is called from within a DLL, routines registered with _onexit run on a DLL's unloading after DllMain is called with DLL_PROCESS_DETACH.
https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/onexit-onexit-m?view=msvc-160

We'll probably need to go a bit deeper.

[justanotheranonymoususer]

I think that it's only relevant for static linkage. Dynamic linkage msvcrt.dll can't know who the caller is, since there's only one msvcrt.dll in the process address space.

I replaced _onexit with atexit at linking stage (didn't want to recompile the code), and it fixed the problem for me.

[justanotheranonymoususer]

The command line switches for the workaround:
32-bit:
-Xlinker --defsym=__onexit=_atexit
64-bit:
-Xlinker --defsym=_onexit=atexit