Revert "[compiler-rt] Avoid generating coredumps when piped to a tool"

[antmox]

This reverts commit 27e5312.

This commit broke some bots:

• clang-aarch64-sve-vla https://lab.llvm.org/buildbot/#/builders/197/builds/13609
• clang-aarch64-sve-vls https://lab.llvm.org/buildbot/#/builders/184/builds/10988
• clang-aarch64-lld-2stage https://lab.llvm.org/buildbot/#/builders/185/builds/6312

#83701

[llvmbot]

@llvm/pr-subscribers-compiler-rt-sanitizer

Author: antoine moynault (antmox)

Changes

This reverts commit 27e5312.

This commit broke some bots:

• clang-aarch64-sve-vla https://lab.llvm.org/buildbot/#/builders/197/builds/13609
• clang-aarch64-sve-vls https://lab.llvm.org/buildbot/#/builders/184/builds/10988
• clang-aarch64-lld-2stage https://lab.llvm.org/buildbot/#/builders/185/builds/6312

#83701

---

Full diff: https://github.com/llvm/llvm-project/pull/85390.diff

2 Files Affected:

• (modified) compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp (+1-18)
• (modified) compiler-rt/test/sanitizer_common/TestCases/corelimit.cpp (+1-6)

diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp b/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp
index 3605d0d666e3fd..ef1fc354974396 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cpp
@@ -104,24 +104,7 @@ static void setlim(int res, rlim_t lim) {
 
 void DisableCoreDumperIfNecessary() {
   if (common_flags()->disable_coredump) {
-    rlimit rlim;
-    CHECK_EQ(0, getrlimit(RLIMIT_CORE, &rlim));
-    // On Linux, if the kernel.core_pattern sysctl starts with a '|' (i.e. it
-    // is being piped to a coredump handler such as systemd-coredumpd), the
-    // kernel ignores RLIMIT_CORE (since we aren't creating a file in the file
-    // system) except for the magic value of 1, which disables coredumps when
-    // piping. 1 byte is too small for any kind of valid core dump, so it
-    // also disables coredumps if kernel.core_pattern creates files directly.
-    // While most piped coredump handlers do respect the crashing processes'
-    // RLIMIT_CORE, this is notable not the case for Debian's systemd-coredump
-    // due to a local patch that changes sysctl.d/50-coredump.conf to ignore
-    // the specified limit and instead use RLIM_INFINITY.
-    //
-    // The alternative to using RLIMIT_CORE=1 would be to use prctl() with the
-    // PR_SET_DUMPABLE flag, however that also prevents ptrace(), so makes it
-    // impossible to attach a debugger.
-    rlim.rlim_cur = Min<rlim_t>(SANITIZER_LINUX ? 1 : 0, rlim.rlim_max);
-    CHECK_EQ(0, setrlimit(RLIMIT_CORE, &rlim));
+    setlim(RLIMIT_CORE, 0);
   }
 }
 
diff --git a/compiler-rt/test/sanitizer_common/TestCases/corelimit.cpp b/compiler-rt/test/sanitizer_common/TestCases/corelimit.cpp
index fed2e1d89cbffa..2378a4cfdced12 100644
--- a/compiler-rt/test/sanitizer_common/TestCases/corelimit.cpp
+++ b/compiler-rt/test/sanitizer_common/TestCases/corelimit.cpp
@@ -10,12 +10,7 @@ int main() {
   getrlimit(RLIMIT_CORE, &lim_core);
   void *p;
   if (sizeof(p) == 8) {
-#ifdef __linux__
-    // See comments in DisableCoreDumperIfNecessary().
-    assert(lim_core.rlim_cur == 1);
-#else
-    assert(lim_core.rlim_cur == 0);
-#endif
+    assert(0 == lim_core.rlim_cur);
   }
   return 0;
 }

[MaskRay]

@arichardson compiler-rt/test/sanitizer_common/TestCases/corelimit.cpp needs an update

[arichardson]

@arichardson compiler-rt/test/sanitizer_common/TestCases/corelimit.cpp needs an update

Hmm sounds like rlimit was already zero, will need to relax the test to accept zero or one to handle the min().