Bring back stack tracing

[wyldckat]

Back in 2011, we were able to add a simple stack trace mechanism, by using a library that had several stack tracing excerpts from online projects. For example, seen here: https://github.com/blueCFD/ThirdParty-2.1.x/tree/blueCFD-2.1-2/stack_trace

Unfortunately, it after the release of blueCFD(-Core) 2.1-1, subsequent releases had the stack tracer broken for 64-bit releases. It was really weird, but apparently something either changed on our building infrastructure or something was patched on Windows.

The objectives for this task originally seemed to be the following:

1. As suggested in one of the comments below, namely relying on GDB to give a good stack trace.
2. Detection of GDB is a must, so that GDB won't be called unnecessarily.
3. Can keep the current global detection of whether to do bug tracing or not.
4. Additionally, the ability to debug just-on-time would also be very helpful. At the very least, provide the user with the ability to debug the crash, by indicating the command to run on another terminal, while the application sleeps until terminated by GDB, as well as the command for terminating it if GDB is not to be used.

Nonetheless, the comments below offer more details.

[wyldckat]

As of OpenFOAM 2.2.0, they have got the safePrintStack, which uses embedded debugging. The only way we could do this in Windows would be to embed GDB code :(

[wyldckat]

Research done so far

• Old stacktrace project: http://sourceforge.net/p/stacktrace/code/HEAD/tree/
• StackWalker: http://stackwalker.codeplex.com/releases/view/109444
  • Doesn't work.
• Simple stack tracing: http://stackoverflow.com/questions/5693192/win32-backtrace-from-c-code
  • Doesn't work.
• Calling GDB from inside.

  • Doesn't work. Or at least, using the following is not enough:

    void error::printStack(Ostream& os)
    {
        std::ostringstream test;
        TCHAR szFileName[MAX_PATH];
        GetModuleFileName( NULL, szFileName, MAX_PATH );
    
        test << "gdb " << szFileName << " " << pid();
        system(test.str());
    }
    

    It seems like the call to system tampers with the stack tracing capabilities.

[wyldckat]

Determined possible reasons why it's not working properly on 64bit

It looks like there is a very big flaw in Windows XP to 7 64-bit in handling CPUs with AVX, along with there being a frame hiding maneuver that seems to only be possible on x86_64, which makes the call stack walking a major problem. For more information on this topic, see these two blog posts:

• http://randomascii.wordpress.com/2012/07/22/more-adventures-in-failing-to-crash-properly/
• http://randomascii.wordpress.com/2012/07/05/when-even-crashing-doesnt-work/

This is a very good reference of several details on call stack and debuggers:

• http://devonstrawntech.tumblr.com/post/15878429193/how-to-write-a-windows-debugger-references

[wyldckat]

About MSys2

It's possible that MSys2's software stack can help sort out this issue, given that it has utilities that OpenFOAM uses on Linux for stack tracing as well, namely the addr2line utility that is part of binutils.
Therefore, we might be able to use at least part of OpenFOAM's way of stack tracing.

[wyldckat]

It seems that perhaps we could use this library instead: http://boostorg.github.io/stacktrace/stacktrace/configuration_and_build.html

It looks like we can follow these installation instructions: boostorg/stacktrace#14 (comment)

[wyldckat]

Commands used to try out the boostorg/stacktrace option:

foam3rdParty
git clone https://github.com/ianlancetaylor/libbacktrace.git
mkdir -p build/libbacktrace
cd build/libbacktrace
../../libbacktrace/configure --prefix /home/ofuser/blueCFD/ThirdParty-5.x/platforms/mingw_w64GccDPInt32/
make
make install

foam3rdParty
git clone https://github.com/boostorg/stacktrace.git

Then for testing:

cd stacktrace/example
g++ -I/home/ofuser/blueCFD/ThirdParty-5.x/platforms/mingw_w64GccDPInt32/include -I../include -o assert_handler_basic.exe assert_handler.cpp
./assert_handler_basic.exe
g++  -DBOOST_STACKTRACE_USE_BACKTRACE -I/home/ofuser/blueCFD/ThirdParty-5.x/platforms/mingw_w64GccDPInt32/include -I../include -L/home/ofuser/blueCFD/ThirdParty-5.x/platforms/mingw_w64GccDPInt32/lib -o assert_handler_basic.exe assert_handler.cpp -lbacktrace
./assert_handler_basic.exe

But it's not working :( There is no stack trace.

However, at least the debug_function.cpp example is working... since it only gives a frame listing.

[wyldckat]

On MSys2 i686, it worked well, at least with an old installation of MSys2 that provided GCC 6.3.0.

Am updating it to the more recent MSys2 stack, to see what might need fixing.

• Works just fine in i686... even after updating the stack... to GCC 7.2.0 et al...

At least on i686, it was possible so far to assess that using -gdwarf alone when compile is enough for giving enough data for addr2line to work:

g++ -I../include -o assert_handler_basic.exe assert_handler.cpp -gdwarf

./assert_handler_basic.exe

Results in:

Expression 'i < N' is false in function 'T& boost::array<T, N>::operator[](boost::array<T, N>::size_type) [with T = int; unsigned int N = 5; boost::array<T, N>::reference = int&; boost::array<T, N>::size_type = unsigned int]': out of range.
Backtrace:
 0# 0x00403BA0 in C:\msys32\home\ofuser\stacktrace\example\assert_handler_basic.exe
 1# 0x00403DF7 in C:\msys32\home\ofuser\stacktrace\example\assert_handler_basic.exe
 2# 0x0040163F in C:\msys32\home\ofuser\stacktrace\example\assert_handler_basic.exe
 3# 0x00401671 in C:\msys32\home\ofuser\stacktrace\example\assert_handler_basic.exe
 4# 0x0040164C in C:\msys32\home\ofuser\stacktrace\example\assert_handler_basic.exe
 5# 0x00401671 in C:\msys32\home\ofuser\stacktrace\example\assert_handler_basic.exe
 6# 0x0040168E in C:\msys32\home\ofuser\stacktrace\example\assert_handler_basic.exe
 7# 0x004013EB in C:\msys32\home\ofuser\stacktrace\example\assert_handler_basic.exe

Which we can then do:

addr2line.exe 0x00403BA0 -e assert_handler_basic.exe

Which gave:

C:\msys32\home\ofuser\stacktrace\example/../include/boost/stacktrace/stacktrace.hpp:130

[wyldckat]

After considerable reading into pull requests and reports on https://github.com/rust-lang/rust, the assessment so far is that perhaps the issue on x64 is related to problems on either the executable path is retrieved and provided to the libstacktrace library, or there is some memory corruption midway the backtrace.

[wyldckat]

Playing around with the mixing of debug_function.cpp and assert_handler.cpp, it was possible to assess that the boost::stacktrace::frame gives use the ability to get the memory address of a particular function... although it doesn't seem very useful.

[wyldckat]

Using the example https://github.com/ErwanLegrand/libbacktrace/blob/master/examples/bt.c - it was possible to diagnose that the problem is on the x64 builds, because it works just fine on i686.

Clearly the unwrapper is broken, because it gives 7 entries on each category on x64, versus 3 on i686:

• x64:

0xffffffff ???
        ???:0
0xffffffff ???
        ???:0
0xffffffff ???
        ???:0
0xffffffff ???
        ???:0
0xffffffff ???
        ???:0
0xffffffff ???
        ???:0
0xffffffff ???
        ???:0
ffffffff ?? ??:0
ffffffff ?? ??:0
ffffffff ?? ??:0
ffffffff ?? ??:0
ffffffff ?? ??:0
ffffffff ?? ??:0
ffffffff ?? ??:0

• i686:

0x401787 bt
        C:\msys32\home\ofuser/bt.c:51
0x4017f7 main
        C:\msys32\home\ofuser/bt.c:58
0x4013ea __tmainCRTStartup
        C:/repo/mingw-w64-crt-git/src/mingw-w64/mingw-w64-crt/crt/crtexe.c:341
4017b1 (null) bt:12389752
4017f7 (null) main:12389752
4013ea (null) __tmainCRTStartup:12413960

[wyldckat]

Another more detailed test is the one done by btest.exe in libbacktrace. To build it:

make btest.exe

Then running it will give the following output:

• On i686:

./btest.exe
PASS: backtrace_full noinline
PASS: backtrace_full inline
PASS: backtrace_simple noinline
PASS: backtrace_simple inline

• On x86_64:

FAIL: backtrace_full noinline
FAIL: backtrace_full inline
FAIL: backtrace_simple noinline
FAIL: backtrace_simple inline
test1: [0]: missing file name or function name
test2: [0]: missing file name or function name
test3: [0]: missing file name or function name
test3: [0]: NULL syminfo name
test3: [1]: NULL syminfo name
test3: [2]: NULL syminfo name
test4: [0]: missing file name or function name

[wyldckat]

In an attempt to try and build Rust's libbacktrace implementation, I stumbled upon this report: msys2/MSYS2-packages#735 (comment) - due to a bug in how git submodule works on MSys2 in MinGW64 mode, which will give an error message similar to the following:

': not a valid identifierline 88: export: `dashless

The solution was to hack the file /mingw64/bin/gettext.sh and replace the functions eval_gettext and eval_ngettext with the following chunk of code:

if [ -e /usr/bin/envsubst ]
then
    ENVSUBST=/usr/bin/envsubst
else
    ENVSUBST=envsubst
fi

# eval_gettext MSGID
# looks up the translation of MSGID and substitutes shell variables in the
# result.
eval_gettext () {
  gettext "$1" | (export PATH `$ENVSUBST --variables "$1"`; envsubst "$1")
}

# eval_ngettext MSGID MSGID-PLURAL COUNT
# looks up the translation of MSGID / MSGID-PLURAL for COUNT and substitutes
# shell variables in the result.
eval_ngettext () {
  ngettext "$1" "$2" "$3" | (export PATH `$ENVSUBST --variables "$1 $2"`; envsubst "$1 $2")
}

On a side-note: using libstacktrace.a as built by the rust stack, didn't solve the issue. Which by the way, I got by running:

./x.py build --stage 0

[wyldckat]

The somewhat good news is that it is possible to generate stack traces in 64-bit builds, as proven by this blog post: http://theorangeduck.com/page/printing-stack-trace-mingw

I've tested with success by using the MSys2 stack being used for blueCFD-Core 2017-1, however:

1. We still have to compile with the option -gdwarf, so that it provides the necessary function data.
2. Had to use the pre-built cv2pdb application that is available here: https://github.com/rainers/cv2pdb/releases
3. The option -Wl,--strip-all cannot be used, not even -Wl,--strip-debug.

Furthermore, the other downside is that it doesn't state the line of the crash, although it at least gives the names of the functions which were called.

Furthermore, the following mailing list thread on mingw-w64 has a few more details which may be useful: https://sourceforge.net/p/mingw-w64/mailman/message/26662488/

The downside to using -g or -gdwarf is that the binary files will become considerably larger...

On the other hand, gdb doesn't need the symbols in order to get the mangled names of the functions...

[wyldckat]

Have tried bringing back the libstack_trace we last had in blueCFD(-Core) 2.1, but the stack trace is rather crippled:

• When built as an external library:

Backtrace:
        ZN10StackTraceC1Ev [0x652415a2+66]
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libstack_trace.dll
        (No symbol) [0x622490]
        RtlCaptureContext [0x7ffa93621d00+0]
                 module: [unknown module]
        (No symbol) [0x1c8db0]
        (No symbol) [0x60fc80]
        RtlFreeMemoryBlockLookaside [0x7ffa93fc08f0+6032]
                 module: [unknown module]
        (No symbol) [0x40]

• When the library is statically linked directly:

Backtrace:
        mainCRTStartup [0x4017d2+706]
                 module: X:\ThirdParty-5.x\sandbox\our_stack\ourstack.exe
        (No symbol) [0xe58d70]
        RtlCaptureContext [0x7ffa93621d00+0]
                 module: [unknown module]
        (No symbol) [0xe58980]
        (No symbol) [0x60fc80]
        (No symbol) [0x1]
        RtlAllocateHeap [0x7ffa93ede5a1+14241]
                 module: [unknown module]
        (No symbol) [0xba000100]
        (No symbol) [0x101]
        (No symbol) [0x23]
        (No symbol) [0x118]
        (No symbol) [0x1f800010000f]
        (No symbol) [0x53002b002b0033]
        (No symbol) [0x206002b002b]
        RtlAllocateHeap [0x7ffa93ede57a+14202]
                 module: [unknown module]
        (No symbol) [0x1]
        (No symbol) [0x101]
        (No symbol) [0x26]
        (No symbol) [0x130]

And this was done using a minimal application code, while using OpenFOAM's wmake stack. And this was while using the cv2pdb application.

However, I did not account for the changes made in Chromium:

• https://cs.chromium.org/chromium/src/base/debug/stack_trace.h
• https://cs.chromium.org/chromium/src/base/debug/stack_trace_win.cc

---

Have tried the code modifications in the Chomium code and it has gotten fairly worse:

Backtrace:
        ZN10StackTraceC2Ev [0x0x6524171a+58]

Will try to adapt only specific details from it...

[wyldckat]

Further R&D on adapting our old stack_trace library, has lead us to figure out that:

1. We can use the option -gdwarf, then cv2pdb to extract the debug data, then we can use strip -s to remove the DWARF data.

   • On second note, it looked like cv2pdb already strips the DWARF data.

2. The old code was very unstable and it's weird how it ever worked in the past.

3. I disregarded that Chromium uses CaptureStackBackTrace instead of StackWalk64, which is also stated at http://www.dima.to/blog/?p=13 and a few other StackOverflow threads. This should be used, at least on 64-bit, because the context capturing mechanism does not provide the ability to get a full scope of the context... and without proper context, it won't go all the way back to the start of the process.

4. Have confirmed that if we initialize the context from within the main function of the application, it has a better perspective of the stack trace. However it doesn't give the indication of how it got to the stack walking function in the DLL... therefore, still need to try out CaptureStackBackTrace.

[wyldckat]

Using the following code with our latest libstack_trace.dll:

#include <iostream>
#include <sstream>

#include "stack_trace.h"

static CONTEXT contextRecord;

void stacktraceme(void)
{
    std::stringstream callstacktext
    (
        std::stringstream::in | std::stringstream::out
    );

    StackTrace *traceUs = new StackTrace(&contextRecord);

    if(traceUs!=NULL)
    {
        traceUs->OutputToStream(&callstacktext);

        delete traceUs;
        traceUs=NULL;

        std::cout << callstacktext.str().data();
    }
    else
    {
        std::cout
            << "We are sorry, but the application crashed and stack tracing did "
            << "not work in this specific situation."
            << std::endl;
    }
}

void stacktracemeB(void)
{
    std::stringstream callstacktext
    (
        std::stringstream::in | std::stringstream::out
    );

    StackTrace *traceUs = new StackTrace();

    if(traceUs!=NULL)
    {
        traceUs->OutputToStream(&callstacktext);

        delete traceUs;
        traceUs=NULL;

        std::cout << callstacktext.str().data();
    }
    else
    {
        std::cout
            << "We are sorry, but the application crashed and stack tracing did "
            << "not work in this specific situation."
            << std::endl;
    }
}

void function_c(void) {
  stacktraceme();
}

void function_b(void) {
  function_c();
}

void function_a(void) {
  function_b();
}


int main(int argc, char *argv[]) {
  memset(&contextRecord, 0, sizeof(CONTEXT));
  contextRecord.ContextFlags = CONTEXT_FULL;

  RtlCaptureContext(&contextRecord);

  function_a();

  stacktracemeB();
}

We got the following stack trace:

Backtrace:
        main [0x00403b61+0x31] (ourstack.C:98)
                 module: X:\ThirdParty-5.x\sandbox\our_stack\ourstack.exe


Backtrace:
        __tcf_0 [0x65241465+0x35] (stack_trace_win.C:160)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libstack_trace.dll
        stacktracemeB [0x00401e82+0x1f2] (ourstack.C:81)
                 module: X:\ThirdParty-5.x\sandbox\our_stack\ourstack.exe
        main [0x00403b6b+0x3b] (ourstack.C:119)
                 module: X:\ThirdParty-5.x\sandbox\our_stack\ourstack.exe
        __tmainCRTStartup [0x004013f7+0x247] (C:\repo\mingw-w64-crt-git\src\mingw-w64\mingw-w64-crt\crt\crtexe.c:343)
                 module: X:\ThirdParty-5.x\sandbox\our_stack\ourstack.exe
        mainCRTStartup [0x0040152b+0x1b] (C:\repo\mingw-w64-crt-git\src\mingw-w64\mingw-w64-crt\crt\crtexe.c:221)
                 module: X:\ThirdParty-5.x\sandbox\our_stack\ourstack.exe
        BaseThreadInitThunk [0x7ffe6bca2774+0x14]
                 module: C:\WINDOWS\System32\KERNEL32.DLL
        RtlUserThreadStart [0x7ffe6cc20d51+0x21]
                 module: C:\WINDOWS\SYSTEM32\ntdll.dll

Which means that using CaptureStackBackTrace gives us the best stack trace possible, without having to rely on a hack to provide the context. Furthermore, the intermediate functions are merged by GCC during compilation, which is why they are unwrapped and forgotten.

However, this context option may be useful in some stack tracing scenarios, such as the safe_printstack (if I remember correctly) that OpenFOAM has got.

However, this strategy requires us to:

1. Build our own libstack_trace library and link it to libOpenFOAM, via OSspecific/MSwindows.
2. All applications must be built without the -Wl,--strip-all option.
3. The cv2pdb application must be downloaded and used on all executable builds (i.e. .EXE and .DLL).

For example, not using cv2pdb on the relevant binaries, gave me the following output:

Backtrace:
        (No symbol) [0x403b61]
                 module: X:\ThirdParty-5.x\sandbox\our_stack\ourstack.exe


Backtrace:
        ZN10StackTraceC1Ev [0x65241465+0x25]
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libstack_trace.dll
        (No symbol) [0x401e82]
                 module: X:\ThirdParty-5.x\sandbox\our_stack\ourstack.exe
        (No symbol) [0x403b6b]
                 module: X:\ThirdParty-5.x\sandbox\our_stack\ourstack.exe
        (No symbol) [0x4013f7]
                 module: X:\ThirdParty-5.x\sandbox\our_stack\ourstack.exe
        (No symbol) [0x40152b]
                 module: X:\ThirdParty-5.x\sandbox\our_stack\ourstack.exe
        BaseThreadInitThunk [0x7ffe6bca2774+0x14]
                 module: C:\WINDOWS\System32\KERNEL32.DLL
        RtlUserThreadStart [0x7ffe6cc20d51+0x21]
                 module: C:\WINDOWS\SYSTEM32\ntdll.dll

However, in this scenario, it is possible to retrieve the line via addr2line, e.g.:

addr2line 0x401e82 -e "ourstack.exe"

X:\ThirdParty-5.x\sandbox\our_stack/ourstack.C:81

However, it only works if the DWARF data is not stripped out.

[wyldckat]

Will go forward with this solution for blueCFD-Core 2017. The development steps to wrap this up are as follows:

1. Finish revising the descriptions in our libstack_trace.
2. Add the LICENSE file from Chromium to that folder.
3. Add a new AllwmakeMinGWStackTrace script, which will download cv2pdb and build libstack_trace. Check the other similar scripts for details.
4. Call this script from within ThirdParty-5.x/Allwmake.
5. Remove -Wl,strip-all and add -gdwarf to the related rules files.
6. Revise the stack tracing functions in OSspecific/MSwindows.
7. Revise the file src/OpenFOAM/Make/options.
8. Need to add cv2pdb to the search paths.
   • Actually, haven't added it to the PATH variable and instead will use the variable CV2PDB_PATH to be able to know when it's possible to use it or not.
9. Need to add an extra command line to the wmake/makefiles/general for calling cv2pdb.
10. Test rebuilding the main libraries, namely OSspecific/MSwindows and libOpenFOAM, then test how stack tracing is working.
11. Sync up with the upstream OpenFOAM 5.x and rebuild our port.

For point 10, here is the example output:

Backtrace:
        ZN10StackTraceC1Ev [0x705c1465+0x25]
                 module: X:\ThirdParty-5.x\platforms\mingw_w64GccDPInt32\lib\libstack_trace.dll
        printStack [0x10e123a+0x1fa] (printStack.C:92)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libOpenFOAM.dll
        sigFpeHandler [0x10e19d3+0x33] (signals\sigFpe.C:79)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libOpenFOAM.dll
        (No symbol) [0x403cad]
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\bin\simpleFoam.exe
        _C_specific_handler [0x7ffa5d486466+0x96]
                 module: C:\WINDOWS\SYSTEM32\ntdll.dll
        0_chkstk [0x7ffa5d49a07d+0x11d]
                 module: C:\WINDOWS\SYSTEM32\ntdll.dll
        RtlLookupFunctionEntry [0x7ffa5d409c58+0xab8]
                 module: C:\WINDOWS\SYSTEM32\ntdll.dll
        KiUserExceptionDispatcher [0x7ffa5d4990fe+0x2e]
                 module: C:\WINDOWS\SYSTEM32\ntdll.dll
        sumProd<double> [0x1086f05+0x1a5] (fields\Fields\scalarField\scalarField.C:114)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libOpenFOAM.dll
        ZN4Foam6divideINS_12fvPatchFieldENS_7volMeshEEEvRNS_14GeometricFieldIdT_T0_EERKS6_S9_ [0x708455e0+0xb0]
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libincompressibleTurbulenceModels.dll
        ZN4FoamdvINS_12fvPatchFieldENS_7volMeshEEENS_3tmpINS_14GeometricFieldIdT_T0_EEEERKS8_RKS7_ [0x7089097f+0x26f]
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libincompressibleTurbulenceModels.dll
        ZN4Foam9RASModels8kEpsilonINS_29IncompressibleTurbulenceModelINS_14transportModelEEEE10correctNutEv [0x70881172+0x52]
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libincompressibleTurbulenceModels.dll
        (No symbol) [0x44dae9]
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\bin\simpleFoam.exe
        (No symbol) [0x4013f7]
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\bin\simpleFoam.exe
        (No symbol) [0x40152b]
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\bin\simpleFoam.exe
        BaseThreadInitThunk [0x7ffa5ce32774+0x14]
                 module: C:\WINDOWS\System32\KERNEL32.DLL
        RtlUserThreadStart [0x7ffa5d460d51+0x21]
                 module: C:\WINDOWS\SYSTEM32\ntdll.dll

With PDB files for all of the relevant modules:

Backtrace:
        0__tcf_0 [0x705c1465+0x35] (stack_trace_win.C:169)
                 module: X:\ThirdParty-5.x\platforms\mingw_w64GccDPInt32\lib\libstack_trace.dll
        printStack [0x10b0c28+0x218] (printStack.C:96)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libOpenFOAM.dll
        sigFpeHandler [0x10b1a93+0x33] (signals\sigFpe.C:79)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libOpenFOAM.dll
        _gnu_exception_handler [0x403cad+0xcd] (C:\repo\mingw-w64-crt-git\src\mingw-w64\mingw-w64-crt\crt\crt_handler.c:270)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\bin\simpleFoam.exe
        _C_specific_handler [0x7ffa5d486466+0x96]
                 module: C:\WINDOWS\SYSTEM32\ntdll.dll
        0_chkstk [0x7ffa5d49a07d+0x11d]
                 module: C:\WINDOWS\SYSTEM32\ntdll.dll
        RtlLookupFunctionEntry [0x7ffa5d409c58+0xab8]
                 module: C:\WINDOWS\SYSTEM32\ntdll.dll
        KiUserExceptionDispatcher [0x7ffa5d4990fe+0x2e]
                 module: C:\WINDOWS\SYSTEM32\ntdll.dll
        sumProd<double> [0x1056f05+0x1a5] (fields\Fields\scalarField\scalarField.C:114)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libOpenFOAM.dll
        _matherr [0x708455e0+0x5e4b0]
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libincompressibleTurbulenceModels.dll
        _matherr [0x7089097f+0xa984f]
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libincompressibleTurbulenceModels.dll
        _matherr [0x70881172+0x9a042]
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libincompressibleTurbulenceModels.dll
        mingw_set_invalid_parameter_handler [0x44dae9+0x49599]
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\bin\simpleFoam.exe
        __tmainCRTStartup [0x4013f7+0x247] (C:\repo\mingw-w64-crt-git\src\mingw-w64\mingw-w64-crt\crt\crtexe.c:343)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\bin\simpleFoam.exe
        mainCRTStartup [0x40152b+0x1b] (C:\repo\mingw-w64-crt-git\src\mingw-w64\mingw-w64-crt\crt\crtexe.c:221)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\bin\simpleFoam.exe
        BaseThreadInitThunk [0x7ffa5ce32774+0x14]
                 module: C:\WINDOWS\System32\KERNEL32.DLL
        RtlUserThreadStart [0x7ffa5d460d51+0x21]
                 module: C:\WINDOWS\SYSTEM32\ntdll.dll

Which isn't exactly entirely accurate, but given that there is a lot of template and macro management being used in the code, it's natural that it's not entirely accurate... it seems that we could use the ability to see both the mangled and unmangled names.

However, after considerable testing, it was concluded that we cannot easily get the mangled/decorated names once we have the PDB files. But this was not a problem, once a full rebuild was done, apparently due to stale object files having been used for the previous output. A clean output for a clean build was this:

Backtrace:
        0__tcf_0 [0x705c1465+0x35] (stack_trace_win.C:169)
                 module: X:\ThirdParty-5.x\platforms\mingw_w64GccDPInt32\lib\libstack_trace.dll
        printStack [0x34060c28+0x218] (printStack.C:96)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libOpenFOAM.dll
        sigFpeHandler [0x34061a93+0x33] (signals\sigFpe.C:79)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libOpenFOAM.dll
        _gnu_exception_handler [0x403cad+0xcd] (C:\repo\mingw-w64-crt-git\src\mingw-w64\mingw-w64-crt\crt\crt_handler.c:270)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\bin\simpleFoam.exe
        _C_specific_handler [0x7ffa5d486466+0x96]
                 module: C:\WINDOWS\SYSTEM32\ntdll.dll
        0_chkstk [0x7ffa5d49a07d+0x11d]
                 module: C:\WINDOWS\SYSTEM32\ntdll.dll
        RtlLookupFunctionEntry [0x7ffa5d409c58+0xab8]
                 module: C:\WINDOWS\SYSTEM32\ntdll.dll
        KiUserExceptionDispatcher [0x7ffa5d4990fe+0x2e]
                 module: C:\WINDOWS\SYSTEM32\ntdll.dll
        sumProd<double> [0x34006f05+0x1a5] (fields\Fields\scalarField\scalarField.C:114)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libOpenFOAM.dll
        divide<Foam::fvPatchField, Foam::volMesh> [0x708455e0+0xb0]
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libincompressibleTurbulenceModels.dll
        operator/<Foam::fvPatchField, Foam::volMesh> [0x7089097f+0x26f] (X:\OpenFOAM-5.x\src\OpenFOAM\containers\Lists\List\List.T.C:271)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libincompressibleTurbulenceModels.dll
        correctNut [0x70881172+0x52] (..\turbulenceModels\lnInclude\..\RAS\kEpsilon\kEpsilon.C:42)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\lib\libincompressibleTurbulenceModels.dll
        0000main [0x44dae9+0x25d9] (simpleFoam.C:57)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\bin\simpleFoam.exe
        __tmainCRTStartup [0x4013f7+0x247] (C:\repo\mingw-w64-crt-git\src\mingw-w64\mingw-w64-crt\crt\crtexe.c:343)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\bin\simpleFoam.exe
        mainCRTStartup [0x40152b+0x1b] (C:\repo\mingw-w64-crt-git\src\mingw-w64\mingw-w64-crt\crt\crtexe.c:221)
                 module: X:\OpenFOAM-5.x\platforms\mingw_w64GccDPInt32Opt\bin\simpleFoam.exe
        BaseThreadInitThunk [0x7ffa5ce32774+0x14]
                 module: C:\WINDOWS\System32\KERNEL32.DLL
        RtlUserThreadStart [0x7ffa5d460d51+0x21]
                 module: C:\WINDOWS\SYSTEM32\ntdll.dll

[wyldckat]

Important note for the future

The build option -gz (see https://gcc.gnu.org/onlinedocs/gcc/Debugging-Options.html) should be used if we switch to a future version of libbacktrace, because a recent test gave a 220MB file libOpenFOAM.dll, because of the DWARF symbol data, versus 21MB when stripped.

However, while we are using cv2pdb to interpret and strip away the DWARF symbol data, might was well not do that for now.

Although it is not clear if the option -gz will actually work on the MinGW-w64 stack, because ld has complained that it doesn't know about --compress-debug-sections=zlib.

Have added this note to #71.

[wyldckat]

This issue is now solved up to the following commits:

• On the port for OpenFOAM-5.x: commit d987cc8e28c519d9985d04e155d58fb82780c43d
• On the port for ThirdParty-5.x: commit cf1f2732ca50cab8b175cc2914cc7f199523f930

[wyldckat]

There is a somewhat critical issue with how cv2pdb operates on libfiniteVolume.dll: it is able to generate a libfiniteVolume.pdb file, however it is not able to properly strip the file down.

This means that the libfiniteVolume.dll file with DWARF data in it, has a size of 828MB, while it has 22MB in blueCFD-Core 2016.

Therefore, what we should do is run strip -s in case cv2pdb fails to work properly.

[wyldckat]

Fixed in commit 01be2c05866b9b07ae2b04d74274ec2561c752eb, even though this still means that cv2pdb is failing to work properly with libfiniteVolume.dll.

This specific problem with cv2pdb has been reported here: rainers/cv2pdb#21

[wyldckat]

The bug in cv2pdb has been fixed, as indicated on the aforementioned report on that project. However, the developer gave us the indication that:

• the build of cv2pdb that we are using from AppVeyor is 32-bit;
• that cv2pdb will need twice the memory space for holding the image;
• for larger files, we will need to build the 64-bit version ourselves.

Therefore it's expected that DLL files larger than 1GB will give us a problem with using the 32-bit version of cv2pdb, therefore we will have to build the 64-bit version ourselves if we need such a feature.

Have revised the error/warning message that we shown in wmake cv2pdb fails to work, in commit af731ae1778e662c8b2779a81be24c9c44cf2e0d.