msvc: enable asan compat

[shuffle2]

Since VS 16.8, ASAN support is very usable and easy to enable. This commit adds the build configuration settings to use ASAN, you just need to uncomment the EnableASAN line in Source/VSProps/Configuration.Base.props.

Here's an example of introducing a simple out of bounds access into the dolphin bootup code:

c:\src\dolphin>Binary\x64\DolphinNoGUI.exe "e:\gamecube\Animal Crossing NTSC-U.iso"
=================================================================
==59956==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x12012d609bd0 at pc 0x7ff6eaaf2fd4 bp 0x005f9b6fe3b0 sp 0x005f9b6fe3b0
WRITE of size 4 at 0x12012d609bd0 thread T0
    #0 0x7ff6eaaf2fd3 in BootManager::BootCore(class std::unique_ptr<struct BootParameters,struct std::default_delete<struct BootParameters> >,struct WindowSystemInfo const &) c:\src\dolphin\Source\Core\Core\BootManager.cpp:234
    #1 0x7ff6ea93fe2f in main c:\src\dolphin\Source\Core\DolphinNoGUI\MainNoGUI.cpp:239
    #2 0x7ff6eaee1af3 in __scrt_common_main_seh d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #3 0x7ffd64dc7033  (C:\WINDOWS\System32\KERNEL32.DLL+0x180017033)
    #4 0x7ffd6519d0d0  (C:\WINDOWS\SYSTEM32\ntdll.dll+0x18004d0d0)

0x12012d609bd0 is located 0 bytes to the right of 400-byte region [0x12012d609a40,0x12012d609bd0)
allocated by thread T0 here:
    #0 0x7ffd0dbf9d42  (C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\bin\HostX64\x64\clang_rt.asan_dynamic-x86_64.dll+0x180059d42)
    #1 0x7ff6eaaf2fc4 in BootManager::BootCore(class std::unique_ptr<struct BootParameters,struct std::default_delete<struct BootParameters> >,struct WindowSystemInfo const &) c:\src\dolphin\Source\Core\Core\BootManager.cpp:233
    #2 0x7ff6ea93fe2f in main c:\src\dolphin\Source\Core\DolphinNoGUI\MainNoGUI.cpp:239
    #3 0x7ff6eaee1af3 in __scrt_common_main_seh d:\agent\_work\63\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #4 0x7ffd64dc7033  (C:\WINDOWS\System32\KERNEL32.DLL+0x180017033)
    #5 0x7ffd6519d0d0  (C:\WINDOWS\SYSTEM32\ntdll.dll+0x18004d0d0)

SUMMARY: AddressSanitizer: heap-buffer-overflow c:\src\dolphin\Source\Core\Core\BootManager.cpp:234 in BootManager::BootCore(class std::unique_ptr<struct BootParameters,struct std::default_delete<struct BootParameters> >,struct WindowSystemInfo const &)
Shadow bytes around the buggy address:
  0x0419530c1320: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0419530c1330: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa
  0x0419530c1340: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0419530c1350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0419530c1360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0419530c1370: 00 00 00 00 00 00 00 00 00 00[fa]fa fa fa fa fa
  0x0419530c1380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0419530c1390: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0419530c13a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0419530c13b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0419530c13c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==59956==ABORTING

from:

diff --git a/Source/Core/Core/BootManager.cpp b/Source/Core/Core/BootManager.cpp
index a339ee80dd..e1b2763965 100644
--- a/Source/Core/Core/BootManager.cpp
+++ b/Source/Core/Core/BootManager.cpp
@@ -230,6 +230,9 @@ static GPUDeterminismMode ParseGPUDeterminismMode(const std::string& mode)
 // Boot the ISO or file
 bool BootCore(std::unique_ptr<BootParameters> boot, const WindowSystemInfo& wsi)
 {
+  auto boom = new int[100];
+  boom[100] = 1;
+
   if (!boot)
     return false;

For the GUI version, you should have a debugger like windbg or VS attached, however you can also setup asan to write a dump file.

[iwubcode]

LGTM (untested)

[iwubcode]

I'm not sure if I was doing anything wrong but for anyone's future reference, I was not able to get past some exceptions that seemed to occur when using Qt to open a file. Luckily, all the times I needed to do that I was able to use drag/drop support but it was a little strange.

I also ended up disabling fastmem in Dolphin.ini as I was getting some behavior in Memtools when using a debug build.

[shuffle2]

@iwubcode maybe you need to set your debugger to not catch access violation exceptions or update the debugger. see https://devblogs.microsoft.com/cppblog/asan-for-windows-x64-and-debug-build-support/ (Debugging – Exceptions (on 16.9 Preview 3 and earlier)). You need to do this for fastmem in any case. Not sure about the qt stuff.

[iwubcode]

Thanks @shuffle2 . Actually debugger was up to date and I had all exceptions turned off (not initially but I found that article after my initial issues).

I will say, it was very nice once I resolved those issues. It took me straight to the issue!