LLVM ERROR: out of memory

[grmblfrz]

windows 10 with latest patchset. LLVM 15.0.5

Stack dump:
0.	Program arguments: C:/msys64/clang64/bin/ld.lld -m i386pep -Bdynamic -o cmTC_d9c21.exe C:/msys64/clang64/lib/crt2.o C:/msys64/clang64/lib/crtbegin.o -LC:/msys64/clang64/x86_64-w64-mingw32/lib -LC:/msys64/clang64/x86_64-w64-mingw32/mingw/lib -LC:/msys64/clang64/lib -LC:/msys64/clang64/x86_64-w64-mingw32/sys-root/mingw/lib -LC:/msys64/clang64/lib/clang/15.0.5/lib/windows CMakeFiles/cmTC_d9c21.dir/testCCompiler.c.obj --out-implib libcmTC_d9c21.dll.a --major-image-version 0 --minor-image-version 0 -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -lmingw32 C:/msys64/clang64/lib/clang/15.0.5/lib/windows/libclang_rt.builtins-x86_64.a -lunwind -lmoldname -lmingwex -lmsvcrt -ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 C:/msys64/clang64/lib/clang/15.0.5/lib/windows/libclang_rt.builtins-x86_64.a -lunwind -lmoldname -lmingwex -lmsvcrt -lkernel32 C:/msys64/clang64/lib/crtend.o
Exception Code: 0xC000001D
 #0 0x00007ff8913d4c46 HandleAbort (C:\msys64\clang64\bin\libLLVM-15.dll+0x184c46)
 #1 0x00007ff90f051881 (C:\WINDOWS\System32\ucrtbase.dll+0x71881)
 #2 0x00007ff90f052851 (C:\WINDOWS\System32\ucrtbase.dll+0x72851)
 #3 0x00007ff8912fab25 llvm::report_bad_alloc_error(char const*, bool) (C:\msys64\clang64\bin\libLLVM-15.dll+0xaab25)
 #4 0x00007ff891359178 llvm::SmallVectorBase<unsigned long long>::grow_pod(void*, unsigned long long, unsigned long long) (C:\msys64\clang64\bin\libLLVM-15.dll+0x109178)
 #5 0x00007ff89456bf44 llvm::findVCToolChainViaRegistry(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&, llvm::ToolsetLayout&) (C:\msys64\clang64\bin\libLLVM-15.dll+0x331bf44)
 #6 0x00007ff75320ee7d lld::coff::LinkerDriver::detectWinSysRoot(llvm::opt::InputArgList const&) (C:\msys64\clang64\bin\ld.lld.exe+0x1ee7d)
 #7 0x00007ff753203c8d lld::coff::LinkerDriver::linkerMain(llvm::ArrayRef<char const*>) (C:\msys64\clang64\bin\ld.lld.exe+0x13c8d)
 #8 0x00007ff753202afd lld::coff::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool) (C:\msys64\clang64\bin\ld.lld.exe+0x12afd)
 #9 0x00007ff7532575f3 lld::mingw::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool) (C:\msys64\clang64\bin\ld.lld.exe+0x675f3)
#10 0x00007ff7531f213f llvm::raw_ostream::operator<<(char const*) (C:\msys64\clang64\bin\ld.lld.exe+0x213f)
#11 0x00007ff7531f1823 main (C:\msys64\clang64\bin\ld.lld.exe+0x1823)
#12 0x00007ff7531f1316 WinMainCRTStartup (C:\msys64\clang64\bin\ld.lld.exe+0x1316)
#13 0x00007ff7531f1366 mainCRTStartup (C:\msys64\clang64\bin\ld.lld.exe+0x1366)
#14 0x00007ff90f8774b4 (C:\WINDOWS\System32\KERNEL32.DLL+0x174b4)
#15 0x00007ff9122226a1 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x526a1)
cc: error: linker command failed due to signal (use -v to see invocation)

[llvmbot]

@llvm/issue-subscribers-lld-coff

[mstorsjo]

I wonder why it would run out of memory there at that point - is there something in the environment that makes it construct large amounts of data within findVCToolChainViaRegistry?

But anyway, most of that autodetection could (and probably should) be skipped entirely if running in mingw mode (which is the case here).

[DimitryAndric]

The problem is likely that @grmblfrz's setup is such that the substring \Common7\IDE is not found in VSInstallPath, e.g.:

bool findVCToolChainViaRegistry(std::string &Path, ToolsetLayout &VSLayout) {
  std::string VSInstallPath;
  if (getSystemRegistryString(R"(SOFTWARE\Microsoft\VisualStudio\$VERSION)",
                              "InstallDir", VSInstallPath, nullptr) ||
      getSystemRegistryString(R"(SOFTWARE\Microsoft\VCExpress\$VERSION)",
                              "InstallDir", VSInstallPath, nullptr)) {
    if (!VSInstallPath.empty()) {
      SmallString<256> VCPath(StringRef(VSInstallPath.c_str(),
                                        VSInstallPath.find(R"(\Common7\IDE)")));
      sys::path::append(VCPath, "VC");

      Path = std::string(VCPath.str());
      VSLayout = ToolsetLayout::OlderVS;
      return true;
    }
  }
  return false;
}

If the VSInstallPath.find(R"(\Common7\IDE)") call returns npos, which is usually the literal value -1, it will attempt to allocate an impossibly huge SmallString. :-)

I think there should be at least a check if the VSInstallPath.find() fails, and if so don't attempt to allocate a string, but return false right away...

[DimitryAndric]

Review: https://reviews.llvm.org/D140019