[sanitizer_common] Add experimental flag to tweak dlopen(<main program>)

[thurstond]

This introduces an experimental flag 'test_only_replace_dlopen_main_program'.
When enabled, this will replace dlopen(main program,...) with dlopen(NULL,...),
which is the correct way to get a handle to the main program.

This can be useful when ASan is statically linked, since dladdr((void*)pthread_join)
or similar will return the path to the main program.

Note that dlopen(main program,...) never ends well:

• PIE in recent glibc versions (glibc bugzilla 24323), or non-PIE: return an error
• PIE in current GRTE and older glibc: attempt to load the main program
  again, leading to reinitializing ASan and failing to remap the shadow
  memory.

[llvmbot]

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: Thurston Dang (thurstond)

Changes

This introduces an experimental flag 'test_only_replace_dlopen_main_program'.
When enabled, this will replace dlopen(<main program,...> with dlopen(NULL,...),
which is the correct way to get a handle to the main program.

This can be useful when ASan is statically linked, since dladdr((void*)pthread_join)
or similar will return the path to the main program.

Note that dlopen(<main program>,...) never ends well:

• PIE in recent glibc versions (glibc bugzilla 24323), or non-PIE: return an error
• PIE in current GRTE and older glibc: attempt to load the main program
  again, leading to reinitializing ASan and failing to remap the shadow
  memory.

---

Full diff: https://github.com/llvm/llvm-project/pull/71715.diff

2 Files Affected:

• (modified) compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc (+29-1)
• (modified) compiler-rt/lib/sanitizer_common/sanitizer_flags.inc (+6)

diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc b/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc
index 80efaf54a0607f6..0f7d9da9d148443 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_common_interceptors.inc
@@ -6304,10 +6304,38 @@ INTERCEPTOR(int, fclose, __sanitizer_FILE *fp) {
 #endif
 
 #if SANITIZER_INTERCEPT_DLOPEN_DLCLOSE
+// Returns 1 if key is a suffix of str, 0 otherwise
+static int internal_strcmp_suffix(const char *key, const char *str) {
+  if (!key || !str)
+    return 0;
+
+  if (internal_strlen(key) > internal_strlen(str))
+    return 0;
+
+  return !internal_strcmp(str + internal_strlen(str) - internal_strlen(key),
+                          key);
+}
+
+#  if SANITIZER_GLIBC
+extern char *__progname;
+#  endif
+
 INTERCEPTOR(void*, dlopen, const char *filename, int flag) {
   void *ctx;
   COMMON_INTERCEPTOR_ENTER_NOIGNORE(ctx, dlopen, filename, flag);
-  if (filename) COMMON_INTERCEPTOR_READ_STRING(ctx, filename, 0);
+
+  if (filename) {
+#  if SANITIZER_GLIBC
+    if (common_flags()->test_only_replace_dlopen_main_program &&
+        internal_strcmp_suffix(__progname, filename)) {
+      VPrintf(1, "dlopen interceptor: replacing %s because it matches %s\n",
+              filename, __progname);
+      filename = (char *)0;  // RTLD_DEFAULT
+    }
+#  endif
+    COMMON_INTERCEPTOR_READ_STRING(ctx, filename, 0);
+  }
+
   void *res = COMMON_INTERCEPTOR_DLOPEN(filename, flag);
   Symbolizer::GetOrInit()->InvalidateModuleList();
   COMMON_INTERCEPTOR_LIBRARY_LOADED(filename, res);
diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_flags.inc b/compiler-rt/lib/sanitizer_common/sanitizer_flags.inc
index 6148ae56067cae0..949bdbd148b6b89 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_flags.inc
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_flags.inc
@@ -269,3 +269,9 @@ COMMON_FLAG(bool, detect_write_exec, false,
 COMMON_FLAG(bool, test_only_emulate_no_memorymap, false,
             "TEST ONLY fail to read memory mappings to emulate sanitized "
             "\"init\"")
+// With static linking, dladdr((void*)pthread_join) or similar will return the
+// path to the main program. This flag will replace dlopen(<main program,...>
+// with dlopen(NULL,...), which is the correct way to get a handle to the main
+// program.
+COMMON_FLAG(bool, test_only_replace_dlopen_main_program, false,
+            "TEST ONLY replace dlopen(<main program>,...) with dlopen(NULL)")

[thurstond]

@vitalybuka Latest patch set added a test case: https://github.com/llvm/llvm-project/pull/71715/files#diff-d1de99adc954c53c720052134ed0f36db3ab9b3349dea742896e97c8187cea0c

[kstoimenov]

LGTM

[thurstond]

Will refactor the interceptor so that the extra code is a no-op if the flag isn't set