[llvm-exegesis] Add thread IDs to subprocess memory names

[boomanaiden154]

This patch adds the thread ID to the subprocess memory shared memory names. This avoids conflicts for downstream consumers that might want to consume llvm-exegesis across multiple threads, which would otherwise run into conflicts due to the same PID running multiple instances.

[llvmbot]

@llvm/pr-subscribers-tools-llvm-exegesis

Author: Aiden Grossman (boomanaiden154)

Changes

This patch adds the thread ID to the subprocess memory shared memory names. This avoids conflicts for downstream consumers that might want to consume llvm-exegesis across multiple threads, which would otherwise run into conflicts due to the same PID running multiple instances.

---

Full diff: https://github.com/llvm/llvm-project/pull/84451.diff

2 Files Affected:

• (modified) llvm/tools/llvm-exegesis/lib/SubprocessMemory.cpp (+14-3)
• (modified) llvm/unittests/tools/llvm-exegesis/X86/SubprocessMemoryTest.cpp (+4-1)

diff --git a/llvm/tools/llvm-exegesis/lib/SubprocessMemory.cpp b/llvm/tools/llvm-exegesis/lib/SubprocessMemory.cpp
index a49fa077257d00..28dc4488a2a004 100644
--- a/llvm/tools/llvm-exegesis/lib/SubprocessMemory.cpp
+++ b/llvm/tools/llvm-exegesis/lib/SubprocessMemory.cpp
@@ -14,6 +14,7 @@
 #ifdef __linux__
 #include <fcntl.h>
 #include <sys/mman.h>
+#include <sys/syscall.h>
 #include <unistd.h>
 #endif
 
@@ -22,12 +23,21 @@ namespace exegesis {
 
 #if defined(__linux__) && !defined(__ANDROID__)
 
+long getCurrentTID() {
+  // We're using the raw syscall here rather than the gettid() function provided
+  // by most libcs for compatibility as gettid() was only added to glibc in
+  // version 2.30.
+  return syscall(SYS_gettid);
+}
+
 Error SubprocessMemory::initializeSubprocessMemory(pid_t ProcessID) {
   // Add the PID to the shared memory name so that if we're running multiple
   // processes at the same time, they won't interfere with each other.
   // This comes up particularly often when running the exegesis tests with
-  // llvm-lit
-  std::string AuxiliaryMemoryName = "/auxmem" + std::to_string(ProcessID);
+  // llvm-lit. Additionally add the TID so that downstream consumers
+  // using multiple threads don't run into conflicts.
+  std::string AuxiliaryMemoryName = "/" + std::to_string(getCurrentTID()) +
+                                    "auxmem" + std::to_string(ProcessID);
   int AuxiliaryMemoryFD = shm_open(AuxiliaryMemoryName.c_str(),
                                    O_RDWR | O_CREAT, S_IRUSR | S_IWUSR);
   if (AuxiliaryMemoryFD == -1)
@@ -47,7 +57,8 @@ Error SubprocessMemory::addMemoryDefinition(
     pid_t ProcessPID) {
   SharedMemoryNames.reserve(MemoryDefinitions.size());
   for (auto &[Name, MemVal] : MemoryDefinitions) {
-    std::string SharedMemoryName = "/" + std::to_string(ProcessPID) + "memdef" +
+    std::string SharedMemoryName = "/" + std::to_string(ProcessPID) + "t" +
+                                   std::to_string(getCurrentTID()) + "memdef" +
                                    std::to_string(MemVal.Index);
     SharedMemoryNames.push_back(SharedMemoryName);
     int SharedMemoryFD =
diff --git a/llvm/unittests/tools/llvm-exegesis/X86/SubprocessMemoryTest.cpp b/llvm/unittests/tools/llvm-exegesis/X86/SubprocessMemoryTest.cpp
index c07ec188a602c4..7c23e7b7e9c5a5 100644
--- a/llvm/unittests/tools/llvm-exegesis/X86/SubprocessMemoryTest.cpp
+++ b/llvm/unittests/tools/llvm-exegesis/X86/SubprocessMemoryTest.cpp
@@ -17,6 +17,7 @@
 #include <endian.h>
 #include <fcntl.h>
 #include <sys/mman.h>
+#include <sys/syscall.h>
 #include <unistd.h>
 #endif // __linux__
 
@@ -49,7 +50,9 @@ class SubprocessMemoryTest : public X86TestBase {
 
   std::string getSharedMemoryName(const unsigned TestNumber,
                                   const unsigned DefinitionNumber) {
-    return "/" + std::to_string(getSharedMemoryNumber(TestNumber)) + "memdef" +
+    long CurrentTID = syscall(SYS_gettid);
+    return "/" + std::to_string(getSharedMemoryNumber(TestNumber)) + "t" +
+           std::to_string(CurrentTID) + "memdef" +
            std::to_string(DefinitionNumber);
   }
 

[boomanaiden154]

This is aimed at enabling the use of multi threads within Gematria, where we want to do memory annotations and benchmarking in parallel to process a large number of snippets (on the order of hundreds of millions now) at a decent rate. There are some other things that additionally need to be added that I have planned for follow-up patches, including core pinning. This patch isn't particularly useful to upstream exegesis, but others, like core pinning, definitely would be.

[github-actions]

✅ With the latest revision this PR passed the C/C++ code formatter.

[fhahn]

It looks like this breaks building at least on macOS, so I went ahead and revert the change for now in aefad27

This is the error I am seeing:

llvm/tools/llvm-exegesis/lib/SubprocessMemory.cpp:146:33: error: out-of-line definition of 'setupAuxiliaryMemoryInSubprocess' does not match any declaration in 'llvm::exegesis::SubprocessMemory'
Expected<int> SubprocessMemory::setupAuxiliaryMemoryInSubprocess(

[dc03-work]

I also got the same error, and it also seems the pre-commit CI was failing with the same error: https://buildkite.com/llvm-project/github-pull-requests/builds/46349#018e31bf-f1a5-4c5e-86f3-47632f4abc15/414-12754.

[boomanaiden154]

Thanks for bringing the issue to my attention and reverting in the meantime. I've relanded it in 8003f55 with a fix.

[truboxl]

When building for Android, I get this

[4027/4315] Linking CXX executable bin/llvm-exegesis
FAILED: bin/llvm-exegesis
: && /home/builder/.termux-build/_cache/android-r26b-api-24-v3/bin/aarch64-linux-android-clang++ -fstack-protector-strong -Oz -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -flto=thin -Os -DNDEBUG -L/data/data/com.termux/files/usr/lib -Wl,-rpath=/data/data/com.termux/files/usr/lib -fopenmp -static-openmp -fno-openmp-implicit-rpath -Wl,--enable-new-dtags -Wl,--as-needed -Wl,-z,relro,-z,now -Wl,--color-diagnostics -flto=thin    -Wl,--gc-sections tools/llvm-exegesis/CMakeFiles/llvm-exegesis.dir/llvm-exegesis.cpp.o -o bin/llvm-exegesis  -Wl,-rpath,"$ORIGIN/../lib"  lib/libLLVMWebAssemblyAsmParser.a  lib/libLLVMAArch64AsmParser.a  lib/libLLVMWebAssemblyCodeGen.a  lib/libLLVMAArch64CodeGen.a  lib/libLLVMWebAssemblyDesc.a  lib/libLLVMAArch64Desc.a  lib/libLLVMWebAssemblyDisassembler.a  lib/libLLVMAArch64Disassembler.a  lib/libLLVMWebAssemblyInfo.a  lib/libLLVMAArch64Info.a  lib/libLLVMCodeGenTypes.a  lib/libLLVMMC.a  lib/libLLVMMCParser.a  lib/libLLVMSupport.a  lib/libLLVMTargetParser.a  -lpthread  lib/libLLVMExegesis.a  lib/libLLVMExegesisAArch64.a  lib/libLLVMWebAssemblyUtils.a  lib/libLLVMWebAssemblyDesc.a  lib/libLLVMWebAssemblyInfo.a  lib/libLLVMAArch64AsmParser.a  lib/libLLVMAArch64CodeGen.a  lib/libLLVMAsmPrinter.a  lib/libLLVMAArch64Disassembler.a  lib/libLLVMAArch64Desc.a  lib/libLLVMAArch64Info.a  lib/libLLVMExegesis.a  lib/libLLVMGlobalISel.a  lib/libLLVMSelectionDAG.a  lib/libLLVMMCDisassembler.a  lib/libLLVMMCA.a  lib/libLLVMObjectYAML.a  lib/libLLVMOrcJIT.a  lib/libLLVMExecutionEngine.a  lib/libLLVMPasses.a  lib/libLLVMCodeGen.a  lib/libLLVMCodeGenTypes.a  lib/libLLVMTarget.a  lib/libLLVMCFGuard.a  lib/libLLVMObjCARCOpts.a  lib/libLLVMCoroutines.a  lib/libLLVMHipStdPar.a  lib/libLLVMipo.a  lib/libLLVMBitWriter.a  lib/libLLVMFrontendOpenMP.a  lib/libLLVMScalarOpts.a  lib/libLLVMAggressiveInstCombine.a  lib/libLLVMInstCombine.a  lib/libLLVMFrontendOffloading.a  lib/libLLVMLinker.a  lib/libLLVMIRPrinter.a  lib/libLLVMVectorize.a  lib/libLLVMInstrumentation.a  lib/libLLVMTransformUtils.a  lib/libLLVMAnalysis.a  lib/libLLVMProfileData.a  lib/libLLVMSymbolize.a  lib/libLLVMDebugInfoDWARF.a  lib/libLLVMDebugInfoPDB.a  lib/libLLVMDebugInfoMSF.a  lib/libLLVMDebugInfoBTF.a  lib/libLLVMJITLink.a  lib/libLLVMOrcTargetProcess.a  lib/libLLVMOrcShared.a  lib/libLLVMWindowsDriver.a  lib/libLLVMOption.a  lib/libLLVMRuntimeDyld.a  lib/libLLVMObject.a  lib/libLLVMMCParser.a  lib/libLLVMMC.a  lib/libLLVMDebugInfoCodeView.a  lib/libLLVMIRReader.a  lib/libLLVMBitReader.a  lib/libLLVMAsmParser.a  lib/libLLVMTextAPI.a  lib/libLLVMAArch64Utils.a  lib/libLLVMCore.a  lib/libLLVMBinaryFormat.a  lib/libLLVMTargetParser.a  lib/libLLVMRemarks.a  lib/libLLVMBitstreamReader.a  lib/libLLVMSupport.a  -lrt  -ldl  -lpthread  -lm  lib/libLLVMDemangle.a && :
ld.lld: error: undefined symbol: llvm::exegesis::SubprocessMemory::getCurrentTID()



referenced by BenchmarkRunner.cpp
lto.tmp:(llvm::exegesis::(anonymous namespace)::SubProcessFunctionExecutorImpl::runWithCounter(llvm::StringRef, llvm::ArrayRef<char const*>, llvm::SmallVectorImpl&) const)
clang-17: error: linker command failed with exit code 1 (use -v to see invocation)
[4028/4315] Linking CXX shared library lib/libLLVM.so.19.0git
ninja: build stopped: subcommand failed.
ERROR: failed to build.

Nvm I fixed by patching llvm/tools/llvm-exegesis/lib/SubprocessMemory.cpp and move out long SubprocessMemory::getCurrentTID() to be guarded by #if defined(__linux__) without !defined(__ANDROID__)

[boomanaiden154]

This should be fixed in a29e85d. If you/others care about that configuration though, you should stand up a buildbot so that we can address these breakages soon after they land rather than a month later.