Address Sanitizer support for native libraries

[rayslava]

We work on sanitization of whole Tizen OS and therefore try to cover as many code as we can with Address Sanitizer.

Currently we came to sanitization of native libraries which are used only by managed applications running on CoreCLR.

Currently we have setup of two parts:

1. Additional build of patched Address Sanitizer run-time with "switchable" interceptors.

2. Patched CoreCLR version with changed native calls generation behavior.

Address Sanitizer patch is required to add additional API to activate or deactivate ASan interception of library calls (e.g. tracing malloc()/free()) in order to get appropriate performance of CoreCLR engine and to avoid reports about CoreCLR internals' issues: our primary customer wants to get reports about their code only, CoreCLR testing is a separate task. Additional build is required since this API introduces a performance overhead to each library call, which we don't want to see for the rest of applications.

CoreCLR version has changes in JIT part in generator of calls from managed code to native libraries, a wrapper is added which calls ASan API to switch tracing on when code from library is executed and turn it off during return back to managed code.

Now we only detect issues inside native libraries, but our next target is annotate CoreCLR allocator, integrate it with ASan and catch bugs for cases when buffer allocated by managed code is passed to native library and corrupted there. Accordingly to our customers, who develop apps for Tizen, this case is rather hard to debug and consequences of such errors are rather bad.

Currently we use GCC to build Tizen OS, so our changes are introduced only to GCC ASan run-time inside libasan.so. Technically it's possible to add it to Clang run-time as well, but it'll require recompilation of target libraries since Clang ASan uses static linkage and we're not sure that someone really needs this support. If you are interested in this kind of testing as well, we could introduce this mode in a conditional build.

cc @alpencolt

[alpencolt]

cc @jkotas

[AaronRobinsonMSFT]

@rayslava It would be helpful to see an example of the kind of change needed. The cost of implementing and maintaining the hooks would determine the acceptance of the work. This is something we should definitely consider and a small example would help quicken that decision on work.

[jkotas]

@rayslava Thank you for writing down this proposal.

If I understand this right, you would like to use the special ASAN allocator when managed code interop with unmanaged code, but keep using the regular allocator for the runtime itself.

Would it make sense to re-reroute all malloc/free done for interop (e.g. the calls to Marshal.AllocHGlobal / Marshal.FreeHGlobal or malloc/free calls done by the interop marshaling layer) to the special ASAN allocator? It would avoid the need for the switching that will be hard to get right 100% of the time, and maybe even avoid a need for a special build of the runtime.

[xiangzhai]

:mips-interest

[dkson95]

Hello @ALL
Recently, I've noticed the related issue on : https://developercommunity.visualstudio.com/t/address-sanitizer-not-working-on-dll-pro/1561504?entry=problem&space=62 .

And, I hope this feature adoption can help to fix and support native libraries(.dll) - memory corruption detection called by dotnet apps.

@rayslava how do you think of above feedback case ?