AVX-512 handling breaks DynamoRIO execution on Windows with full AVX-512 support

[fmoessbauer]

Describe the bug
Execution of any application under DynamoRIO is impossible in the following scenario:

• Windows
• Full AVX-512 Support (i.e. avx512_enabled is set to true in proc.c)
  

  dynamorio/core/arch/x86/proc.c

  Line 437 in cf1ec32

  avx512_enabled = true;

Note: This behavior appears with and w/o client and is deterministic.

To Reproduce
Steps to reproduce the behavior:

1. Take Windows machine with AVX-512 (full) support (No VM), check flag avx512_enabled = true in proc.c.
2. Execute any Program like notepad.exe
   3.1 In release builds, DR immediately exits
   3.2 In debug builds, assertion fires: rip-relative reference out of 32-bit reach

Access Violation, as reported by WinDBG

(3aa8.3a44): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
dynamorio!encode_cti+0x3b [inlined in dynamorio!instr_encode_arch+0x102]:
00000000`7107e042 8803            mov     byte ptr [rbx],al ds:00000000`00000000=??
0:000> kb
 # RetAddr           : Args to Child                                                           : Call Site
00 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : dynamorio!encode_cti+0x3b [c:\users\felix\source\repos\dynamorio\core\arch\x86\encode.c @ 2598] 
01 00000000`7107b13a : 00000000`00000000 00000000`71070c00 00000237`69b0e780 00000000`00000000 : dynamorio!instr_encode_arch+0x102 [c:\users\felix\source\repos\dynamorio\core\arch\x86\encode.c @ 2785] 
02 (Inline Function) : --------`-------- --------`-------- --------`-------- --------`-------- : dynamorio!instr_encode_to_copy+0x16 [c:\users\felix\source\repos\dynamorio\core\arch\encode_shared.c @ 132] 
03 00000000`710b0a51 : 00000000`00000000 00000237`69b0e780 00000000`00000000 00000000`00000002 : dynamorio!instr_encode+0x1a [c:\users\felix\source\repos\dynamorio\core\arch\encode_shared.c @ 140] 
04 00000000`710b1eaa : 00000000`00000007 00000237`69b35b30 00000000`710b32e0 00000000`00000000 : dynamorio!emit_intercept_code+0x1eb1 [c:\users\felix\source\repos\dynamorio\core\win32\callback.c @ 1328] 
05 00000000`710adaf9 : 00000000`7118427f 00000237`69ac1a01 00000000`710b32e0 00000000`00000000 : dynamorio!intercept_call+0x26a [c:\users\felix\source\repos\dynamorio\core\win32\callback.c @ 1743] 
06 00000000`7100594a : 00000000`7118c4b8 00000000`00000218 00000000`00000000 00000000`00000000 : dynamorio!callback_interception_init_finish+0x59 [c:\users\felix\source\repos\dynamorio\core\win32\callback.c @ 7747] 
07 00000000`7109b40c : 000000a1`0510f700 00000000`00000000 00000000`00000000 00000237`67ea0031 : dynamorio!dynamorio_app_init+0x23a [c:\users\felix\source\repos\dynamorio\core\dynamo.c @ 692] 
08 00000000`710d28e0 : 00007ff9`1c1d9770 00000000`710d28d0 00000000`00000000 00000000`00000000 : dynamorio!auto_setup+0x1c [c:\users\felix\source\repos\dynamorio\core\arch\x86_code.c @ 169] 
09 00007ff9`1c1d9770 : 00000000`710d28d0 00000000`00000000 00000000`00000000 00000000`00000000 : dynamorio!dynamo_auto_start+0x10 [C:\Users\felix\CMakeBuilds\1ac36d74-80db-f93a-9dba-3c80b8e1e38c\build\x64-Release\core\x86.asm_core.s @ 1599] 
0a 00000000`710d28d0 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : KERNEL32!GetProcAddressStub
0b 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : dynamorio!dynamo_auto_start [C:\Users\felix\CMakeBuilds\1ac36d74-80db-f93a-9dba-3c80b8e1e38c\build\x64-Release\core\x86.asm_core.s @ 1591] 

Debug Output with loglevel 3

log.0.2376.html.txt

Versions

• All versions newer than (including) cronbuild-7.91.18109
• Prior Versions are not affected
• When manually disabling AVX512 support by setting avx512_enabled = false (in proc.c). Dynamorio is executed correctly

cc @TomSie, @jones718

[derekbruening]

May have been hit in #3923 also.

[derekbruening]

Unfortunately we do not have much Windows automated testing capacity. Help is wanted augmenting that.

[TomSie]

What exactly would you need?

[derekbruening]

Cycles somewhere on Windows with AVX-512 hooked up to Jenkins to run the test suite on pull requests would be ideal. Today we have Appveyor for CI but at least its default does not have AVX-512.

[hgreving2304]

Please read

All non-UNIX 64-bit builds are essentially either not tested or known to be unsupported yet. So far parts of DynamoRIO's code for AVX-512 were active even for those unsupported builds. We are disabling this temporarily in order to get a stable Windows build on AVX-512 machines as pointed out in this issue.

Contributions to get full support in Windows would be greatly appreciated!

[derekbruening]

Thank you for reporting the bug. It should be fixed in the cronbuild at https://github.com/DynamoRIO/dynamorio/releases/tag/cronbuild-7.91.18219

[fmoessbauer]

Thanks for the quick fix. I can confirm that the latest cronbuild works on my AVX512 system (with and without client).