Feature request: Support ARM64

[kmilos]

Available mingw-w64 toolchains for aarch64:

https://github.com/mstorsjo/llvm-mingw/releases
https://www.msys2.org/docs/environments/ (I guess based on/pretty much the same as the above)

[jrfonseca]

I don't object, but I don't have the time nor a setup I could test. I'd happily accept PRs.

[alvinhochun]

It also can't be tested on GitHub Actions (there is no ARM runner available).

[kmilos]

True, this is a bit thinking/preparing in advance 😉

Btw, the Msys2 project has a self-hosted runner if it helps: https://github.com/msys2-arm

There are cca 900 packages available, drmingw being the missing puzzle to try out building e.g darktable and gimp.

[hmartinez82]

So. I tried, the clangarm64 build in MSYS2 is failing:

[59/169] Building CXX object src/common/CMakeFiles/common.dir/log.cpp.obj
FAILED: src/common/CMakeFiles/common.dir/log.cpp.obj
C:\msys64\clangarm64\bin\clang++.exe -DHAVE_WIN64=1 -DPACKAGE_VERSION_MAJOR=0 -DPACKAGE_VERSION_MINOR=9 -DPACKAGE_VERSION_PATCH=3 -DPSAPI_VERSION=1 -DWINVER=0x0601 -D_WIN32_WINNT=0x0601 -IC:/Dev/MINGW-packages/mingw-w64-drmingw/src/drmingw-0.9.3/thirdparty/elf -IC:/Dev/MINGW-packages/mingw-w64-drmingw/src/drmingw-0.9.3/thirdparty/dwarf -IC:/Dev/MINGW-packages/mingw-w64-drmingw/src/drmingw-0.9.3/thirdparty/libiberty -IC:/Dev/MINGW-packages/mingw-w64-drmingw/src/drmingw-0.9.3/thirdparty/zlib -IC:/Dev/MINGW-packages/mingw-w64-drmingw/src/drmingw-0.9.3/include -IC:/Dev/MINGW-packages/mingw-w64-drmingw/src/drmingw-0.9.3/src/common -O2 -pipe -std=gnu++11 -Wall -fno-strict-aliasing -fstack-protector-all -MD -MT src/common/CMakeFiles/common.dir/log.cpp.obj -MF src\common\CMakeFiles\common.dir\log.cpp.obj.d -o src/common/CMakeFiles/common.dir/log.cpp.obj -c C:/Dev/MINGW-packages/mingw-w64-drmingw/src/drmingw-0.9.3/src/common/log.cpp
C:/Dev/MINGW-packages/mingw-w64-drmingw/src/drmingw-0.9.3/src/common/log.cpp:123:34: error: use of undeclared identifier 'CONTEXT_SEGMENTS'
    if (pContext->ContextFlags & CONTEXT_SEGMENTS) {
                                 ^
C:/Dev/MINGW-packages/mingw-w64-drmingw/src/drmingw-0.9.3/src/common/log.cpp:175:46: error: no member named 'Rip' in '_CONTEXT'
        StackFrame.AddrPC.Offset = pContext->Rip;
                                   ~~~~~~~~  ^
C:/Dev/MINGW-packages/mingw-w64-drmingw/src/drmingw-0.9.3/src/common/log.cpp:176:49: error: no member named 'Rsp' in '_CONTEXT'
        StackFrame.AddrStack.Offset = pContext->Rsp;
                                      ~~~~~~~~  ^
C:/Dev/MINGW-packages/mingw-w64-drmingw/src/drmingw-0.9.3/src/common/log.cpp:177:49: error: no member named 'Rbp' in '_CONTEXT'
        StackFrame.AddrFrame.Offset = pContext->Rbp;

This is outside my expertise, it's about the ARM64 framing. I had this same issue when trying to port ProcessHacker/PEView to ARM64 :-(

[jrfonseca]

I don't have the ARM64 expertise neither. From Googling around it does seem that ARM64 also has the concept of frame pointer, so it shouldn't be too difficult to hook it up.

https://docs.microsoft.com/en-us/cpp/build/arm64-windows-abi-conventions?view=msvc-170 describes this but it's not crystal clear.

This is the CONTEXT structure definition for ARM64 from https://sourceforge.net/p/mingw-w64/mingw-w64/ci/master/tree/mingw-w64-headers/include/winnt.h:

  typedef struct DECLSPEC_ALIGN(16) _CONTEXT {
    ULONG ContextFlags;                 /* 000 */
    /* CONTEXT_INTEGER */
    ULONG Cpsr;                         /* 004 */
    union
    {
        struct
        {
            DWORD64 X0;                 /* 008 */
            DWORD64 X1;                 /* 010 */
            DWORD64 X2;                 /* 018 */
            DWORD64 X3;                 /* 020 */
            DWORD64 X4;                 /* 028 */
            DWORD64 X5;                 /* 030 */
            DWORD64 X6;                 /* 038 */
            DWORD64 X7;                 /* 040 */
            DWORD64 X8;                 /* 048 */
            DWORD64 X9;                 /* 050 */
            DWORD64 X10;                /* 058 */
            DWORD64 X11;                /* 060 */
            DWORD64 X12;                /* 068 */
            DWORD64 X13;                /* 070 */
            DWORD64 X14;                /* 078 */
            DWORD64 X15;                /* 080 */
            DWORD64 X16;                /* 088 */
            DWORD64 X17;                /* 090 */
            DWORD64 X18;                /* 098 */
            DWORD64 X19;                /* 0a0 */
            DWORD64 X20;                /* 0a8 */
            DWORD64 X21;                /* 0b0 */
            DWORD64 X22;                /* 0b8 */
            DWORD64 X23;                /* 0c0 */
            DWORD64 X24;                /* 0c8 */
            DWORD64 X25;                /* 0d0 */
            DWORD64 X26;                /* 0d8 */
            DWORD64 X27;                /* 0e0 */
            DWORD64 X28;                /* 0e8 */
            DWORD64 Fp;                 /* 0f0 */
            DWORD64 Lr;                 /* 0f8 */
        } DUMMYSTRUCTNAME;
        DWORD64 X[31];                  /* 008 */
    } DUMMYUNIONNAME;
    /* CONTEXT_CONTROL */
    DWORD64 Sp;                         /* 100 */
    DWORD64 Pc;                         /* 108 */
    /* CONTEXT_FLOATING_POINT */
    NEON128 V[32];                      /* 110 */
    DWORD Fpcr;                         /* 310 */
    DWORD Fpsr;                         /* 314 */
    /* CONTEXT_DEBUG_REGISTERS */
    DWORD Bcr[ARM64_MAX_BREAKPOINTS];   /* 318 */
    DWORD64 Bvr[ARM64_MAX_BREAKPOINTS]; /* 338 */
    DWORD Wcr[ARM64_MAX_WATCHPOINTS];   /* 378 */
    DWORD64 Wvr[ARM64_MAX_WATCHPOINTS]; /* 380 */
  } CONTEXT, *PCONTEXT;

Here are a few examples of invoking StacklWalk64 on ARM64:

• https://hg.mozilla.org/mozilla-central/file/tip/mozglue/misc/StackWalk.cpp
• https://github.com/llvm-mirror/llvm/blob/master/lib/Support/Windows/Signals.inc

Here's an (untested) attempt:

diff --git a/src/common/log.cpp b/src/common/log.cpp
index ef50b82..df170c4 100644
--- a/src/common/log.cpp
+++ b/src/common/log.cpp
@@ -167,17 +167,24 @@ dumpStack(HANDLE hProcess, HANDLE hThread, const CONTEXT *pContext)
     } else {
         // NOLINTNEXTLINE(clang-analyzer-core.NullDereference)
         assert((pContext->ContextFlags & CONTEXT_FULL) == CONTEXT_FULL);
-#ifndef _WIN64
+#if defined(_M_IX86)
         MachineType = IMAGE_FILE_MACHINE_I386;
         dumpContext(pContext);
         StackFrame.AddrPC.Offset = pContext->Eip;
         StackFrame.AddrStack.Offset = pContext->Esp;
         StackFrame.AddrFrame.Offset = pContext->Ebp;
-#else
+#elif defined(_M_AMD64)
         MachineType = IMAGE_FILE_MACHINE_AMD64;
         StackFrame.AddrPC.Offset = pContext->Rip;
         StackFrame.AddrStack.Offset = pContext->Rsp;
         StackFrame.AddrFrame.Offset = pContext->Rbp;
+#elif defined(_M_ARM64)
+        MachineType = IMAGE_FILE_MACHINE_ARM64;
+        StackFrame.AddrPC.Offset = pContext->Pc;
+        StackFrame.AddrStack.Offset = pContext->Sp;
+        StackFrame.AddrFrame.Offset = pContext->Fp;
+#else
+#error Unsupported machine
 #endif
     }
     StackFrame.AddrPC.Mode = AddrModeFlat;

You'll also need to update dumpContext to dump the Arm64 registers, not the AMD64.

There are probably other places in DrMingw source one sees _WIN64 it should probably be split into _M_IX86, _M_AMD64, _M_ARM64

PS: For the record, another alternative I've often considered is to hook https://github.com/ianlancetaylor/libbacktrace into DrMingw. This is far superior to manual frame walking, as it can use the DWARF call stack unwind information. And I suppose it would naturally work on Arm64 too. Alas is a tad too much work than I can afford to invest. I'm also not sure how well it would work unwinding call stack of DLLs build with Microsoft compilers. So not an easy/quick fix.

[hmartinez82]

WIP :)

Dump of file drmingw.exe

PE signature found

File Type: EXECUTABLE IMAGE

FILE HEADER VALUES
            AA64 machine (ARM64)

[jrfonseca]

Cool.

Either StackWalk64 is not returning, or perhaps it's the symbol walking / resolution is failing. You'll need to add a few printfs to narrow the issue down.

If the problem is in DbgHelp.DLL, remember there are usually multiple sources of this DLL: one comes with Windows itself, other comes with WinDBG, so it's worth trying both versions to see if it helps.

Note the registers are 64bits, so you should use %016I64X printf specifier to dump them. We should also wrap around the register dump over multiple text lines.

[hmartinez82]

@jrfonseca I got further. The whole package recipe is compiling in MSYS2 CLANGARM64. Unfortunately I'm working with their version (which is not the latest), because the current MSYS2 patches are really complicated to port.

But for now, I got the code compiling. The changes I've done to the MSYS2 version are the same as these: https://github.com/jrfonseca/drmingw/compare/master...hmartinez82:ClangARM64?expand=1

My current situation is that most of the /tests/apps are crashing drmingw itself once it's launched to handle the exceptions
int3, ud2, nt_assert and a few other are working.

Here's an example of one that is working, int_divide_by_zero.exe: https://pastebin.com/wgNhujH9
But the majority gets stuck before the registers are printed, example from fast_fail or abort_gui or abort_console:

CREATE_PROCESS PID=13284 TID=7152 lpBaseOfImage=00007FF672E70000 abort_gui.exe
LOAD_DLL PID=13284 TID=7152 lpBaseOfDll=00007FFDFBA50000 ntdll.dll
CREATE_THREAD PID=13284 TID=12388
LOAD_DLL PID=13284 TID=7152 lpBaseOfDll=00007FFDFB510000 kernel32.dll
LOAD_DLL PID=13284 TID=7152 lpBaseOfDll=00007FFDF7320000 KernelBase.dll
LOAD_DLL PID=13284 TID=7152 lpBaseOfDll=00007FFDF7B00000 ucrtbase.dll

If I close drmingw itself at that moment, it's launched again trying to handle itself.

DebugView is printing this (even for tests that work):

[16112] DBGHELP: _NT_SYMBOL_PATH: srv*C:\Symbols*http://msdl.microsoft.com/download/symbols  
[16112] DBGHELP: Symbol Search Path: .;srv*C:\Symbols*http://msdl.microsoft.com/download/symbols  
[16112] MGWHELP: libdwarf error - Dwarf_Error is NULL

I wonder if this lack of libdwarf is the issue.

[jrfonseca]

I wonder if the hangs are due to loading symbols from Microsoft symbol server. You can try to set _NT_SYMBOL_PATH=C:\Symbols globally (via Machine settings) or modify the source code to not use symbol server.

Microsoft DLLs such as ucrtbase.dll never have DWARF symbols. By itself, that shouldn't be a problem.

You need to narrow down the issue. The difficulty is lack of tools on Arm64. I think the best approach is adding printfs. I recommend you to add a macro like

#define CHECK_POINT() \
    OutputDebug("drmingw: %s : %u\n", __FILE__, __LINE__)

and sprinkle, entering/leaving functions, first the top level functions, then callees, etc, so you can narrow down which function is being called and not returning.

You can also try using catchsegv instead of drmingw, which allows you to use normal printfs.

Another alternative is tell Clang to generate PDB debug info, and then debug DrMinGW by attaching Windbg.

I also recommend double checking that WinDBG works well as post-mortem debugg. That is, you can also install windbg as post mortem debugger, and you can then try with the samples causing troubles to DrMinGW and confirm that the trouble is truly specific to DrMingw, and not, for example, DbgHelp, or some other Windows component.

[hmartinez82]

I have Visual Studio native in ARM64 running in my machine. I didn't know you could get PDBs from Clang when using the GCC frontend 😮. I'll try that soon.

[kmilos]

Yep: https://blog.llvm.org/2017/08/llvm-on-windows-now-supports-pdb-debug.html

[alvinhochun]

It's noted in the readme of llvm-mingw:

while implemented for aarch64, it doesn't seem to work properly there yet.

Though it may have been improved on git HEAD.

[hmartinez82]

Meanwhile, at least I got the latest drmingw in a PR for MSYS2 msys2/MINGW-packages#11977

[kmilos]

So #70 closes this?

[jrfonseca]

So #70 closes this?

I believe so.

[kmilos]

👍

Not sure if the LLVM 15 release (and imminent packaging for MSYS2) will affect this though:

"Switched the MinGW target to use SEH instead of DWARF for unwind information."

[jrfonseca]

Unfortunately DrMingw has never been capable to unwind stack using MinGW's debugging information. It's been a long standing wish item (which could be achieved, for example, by integrating libbacktrace.) I don't have time myself, but I'd accept PRs if anybody was willing to try.

Be it on Arm or x86, DrMinGW requires -fno-omit-frame-pointer I'm afraid.

[jrfonseca]

@kmilos , @hmartinez82 , FYI, I got Arm binaries generated by GitHub, using llvm-mingw. You can get them from here.

I couldn't test them properly though. I have no Arm hardware. Full system Windows Arm emulation with QEMU is dreadfully slow. I got Windows Arm user mode emulation using QEMU + Arm container plus WINE inside the Arm container, but WINE on ARM's implementation of the debugging API used by drmingw/catchsegv.exe is pretty broken, so I can't actually run any tests.

If you have Arm, could you please give it a try and make sure they are usable?

Known issues:

• These binaries don't include Windows SDK's dbghelp.dll, using system's dbghelp.dll instead, which doesn't support symbol store. That's something to address in the future.
• llvm-rc has stricter parsing, so it doesn't match windres

[alvinhochun]

• llvm-rc has stricter parsing, so it doesn't match windres

I have a feeling I've seen a related bug report somewhere but can't find it. Maybe @mstorsjo knows how to handle this?

[mstorsjo]

(First commenting on a few things I noted mentioned here before.)

It's noted in the readme of llvm-mingw:

while implemented for aarch64, it doesn't seem to work properly there yet.

Though it may have been improved on git HEAD.

Yes, I think it probably has. When I added regression testing for LLDB in llvm-mingw (in mstorsjo/llvm-mingw@6dbfbec) I did test both DWARF and PDB, and it runs fine on i686, x86_64 and aarch64 (while armv7 has a few known issues), at least for the trivial testcase I test there, at least with LLDB (haven't tested it with WinDBG recently). I guess I should update that comment.

Not sure if the LLVM 15 release (and imminent packaging for MSYS2) will affect this though:

"Switched the MinGW target to use SEH instead of DWARF for unwind information."

Actually, this was for the 32 bit ARM target. For ARM64, it has already been using SEH for unwind info (instead of DWARF) since 2019.

• llvm-rc has stricter parsing, so it doesn't match windres

I have a feeling I've seen a related bug report somewhere but can't find it. Maybe @mstorsjo knows how to handle this?

Yeah, this has come up a few times - it has been reported in mstorsjo/llvm-mingw#140, which I fixed in https://reviews.llvm.org/D85183. However, that fix was only for string tables, while MS rc.exe does support string literals concatenated from multiple literal strings in a few more places, and windres supports it essentially anywhere. I think llvm/llvm-project#51286 is about that issue too.

In the fix in https://reviews.llvm.org/D85183 I mentioned that I do have a fix for this, but it felt more risky and messy and hadn't been needed for any code in the wild yet at that point.

If someone wants to have a look, I could rebase my branch with unsubmitted hacks for llvm-rc and publish it somewhere.

[jrfonseca]

Yeah, I think it would be useful if llvm-rc supported the superset of MS rc.exe and windres, to simplify porting. But to be fair, it's not a big deal the application in question here, as the resource file is only used for an about dialog. I imagine most apps nowadays use higher level UI toolkits.

By the way, thanks for your replies and your work on llvm-mingw, @mstorsjo! For many years I've been anticipating for clang-based mingw cross-compilation toolchain, and tried a few different attempts before, but now with llvm-mingw is the first time that it all worked reliably and relatively painlessly.

[hmartinez82]

@jrfonseca It's loading correctly and it's in fact ARM64

[jrfonseca]

Thanks for checking, @hmartinez82. drmingw-x.y.z-win64-arm.7z should have the ARM64, and drmingw-x.y.z-win32-arm.7z the ARM 32bits binaries.