[cxxmodules] Enable the semantic global module index to boost performance.

The global module index (GMI) is an optimization which hides the introduced by
clang overhead when pre-loading the C++ modules at startup.

The GMI represents a mapping between an identifier and a set of modules which
contain this indentifier. This mean that if we TH1 is undeclared the GMI will
load all modules which contain this identifier which is usually suboptimal, too.

The semantic GMI maps identifiers only to modules which contain a definition of
the entity behind the identifier. For cases such as typedefs where the entity
introduces a synonym (rather than declaration) we map the first module we
encounter. For namespaces we add all modules which has a namespace partition.
The namespace case is still suboptimal and further improved by inspecting
what exactly is being looked up in the namespace by the qualified lookup facilities.

Author: vgvassilev

README/README.CXXMODULES.md

@@ -309,10 +309,6 @@ different. There are several differences which can be noticed:
   * rootcling -cxxmodule creates a single artifact *Name.pcm* after the library
   name. At a final stage, ROOT might be able to integrate the Name.pcm with the
   shared library itself.
-  * Preloads all \*pcm files at start up time -- this currently is the only
-  remaining bottleneck which introduces a relatively small performance overhead
-  at startup time and is described bellow. It will be negligible for third-
-  party code (dominated by header parsing).
   * Improved correctness in number of cases -- in a few cases ROOT is more
   correct. In particular, when resolving global variables and function
   declarations which are not part of the ROOT PCH.
@@ -323,6 +319,21 @@ different. There are several differences which can be noticed:
   the LD_LIBRARY_PATH descending to the system libraries. The algorithm is very
   efficient because it uses bloom filters[[5]]. This in turn allows ROOT symbol
   to be extended to system libraries.
+
+### Module Registration Approaches
+
+  The C++ modules system supports /*preloading*/ of all modules at startup time.
+  The current implementation of loading of C++ modules in clang has an overhead
+  and is between 40-60 MB depending on the ROOT configuration while there might
+  be 2x slowdown depending on the workflow. These issues are very likely to be
+  addressed by the LLVM community in midterm.
+
+  Preloading of all C++ modules is semantically the closest to C++ behavior.
+  However, in order to achieve performance ROOT loads them on demand using
+  a global module index file. It has sufficient information to map a looked up
+  identifier to the module which contains the corresponding definition. Switching
+  back to preloading of all C++ modules is done by setting the `ROOT_USE_GMI`
+  environment variable to false.
   
 ### Supported Platforms
 
@@ -349,14 +360,15 @@ different. There are several differences which can be noticed:
 
 ## State of the union
 
-C++ Modules-aware ROOT preloads all modules at start up time. Our motivating
-example:
+Preloading all modules at start up time turn our motivating example into:
 
 ```cpp
 // ROOT prompt
 root [] S *s;           // #1: does not require a definition.
 root [] foo::bar *baz1; // #2: does not require a definition.
 root [] foo::bar baz2;  // #3: requires a definition.
+root [] TCanvas* c = new TCanvas(); // #4 requires a definition
+
 ```
 
 becomes equivalent to
@@ -368,12 +380,29 @@ root [] import Foo.*;
 root [] S *s;           // #1: does not require a definition.
 root [] foo::bar *baz1; // #2: does not require a definition.
 root [] foo::bar baz2;  // #3: requires a definition.
+root [] TCanvas* c = new TCanvas(); // #4 requires a definition
 ```
 
 The implementation avoids recursive actions and relies on a well-defined (by
 the C++ standard) behavior. Currently, this comes with a constant performance
 overhead which we go in details bellow.
 
+ROOT uses the global module index (GMI) to avoid the performance overhead. ROOT
+only preloads the set of C++ modules which are not present in the GMI. The
+example becomes equivalent to:
+
+```cpp
+// ROOT prompt
+root [] import Foo.*;   // Preload Foo if it is not in the GMI.
+root [] S *s;           // #1: does not require a definition.
+root [] foo::bar *baz1; // #2: does not require a definition.
+root [] foo::bar baz2;  // #3: requires a definition.
+root [] TCanvas* c = new TCanvas(); // #4 requires a definition
+```
+
+Line #4 forces cling to send ROOT a callback that TCanvas in unknown but
+the GMI resolves it to module Gpad, loads it and returns the control to cling.
+
 
 ### Performance
 This section compares ROOT PCH technology with C++ Modules which is important but
@@ -385,16 +414,9 @@ is not available.
 The comparisons are to give a good metric when we are ready to switch ROOT to use
 C++ Modules by default. However, since it is essentially the same technology,
 optimizations of C++ Modules also affect the PCH. We have a few tricks up in
-the slaves to but they come with given trade-offs. For example, we can avoid
-preloading of all modules at the cost of introducing recursive behavior in
-loading. This requires to build a global module index which is an on-disk
-hash table. It will contain information about the mapping between an
-identifier and a module name. Upon failed identifier lookup we will use the
-map to decide which set of modules should be loaded. Another optimization
-includes building some of the modules without `-fmodules-local-submodule-visibility`.
-In turn, this would flatten the C++ modules structure and give us performance
-comparable to the ROOT PCH. The trade-off is that we will decrease the
-encapsulation and leak information about implementation-specific header files.
+the sleeves to but they come with given trade-offs.
+
+#### Preloading of C++ Modules
 
 The main focus for the technology preview was not in performance until recently.
 We have invested some resources in optimizations and we would like to show you
@@ -413,6 +435,14 @@ The performance is dependent on many factors such as configuration of ROOT and
 workflow. You can read more at our Intel IPCC-ROOT Showcase presentation
 here (pp 25-33)[[8]].
 
+#### Loading C++ Modules on Demand
+
+In long term, we should optimize the preloading of modules to be a no-op and
+avoid recursive behavior based on identifier lookup callbacks. Unfortunately,
+at the moment the loading of C++ modules on demand shows significantly better
+performance results.
+
+
 You can visit our continuous performance monitoring tool where we compare
 the performance of ROOT against ROOT with a PCH [[9]].
 *Note: if you get error 400, clean your cache or open a private browser session.*

core/dictgen/src/rootcling_impl.cxx

@@ -4332,6 +4332,8 @@ int RootClingMain(int argc,
                                       GetModuleNameFromRdictName(DepMod).str().data());
          }
          DepMod = GetModuleNameFromRdictName(DepMod);
+         // We might deserialize.
+         cling::Interpreter::PushTransactionRAII RAII(&interp);
          if (!interp.loadModule(DepMod, /*complain*/false)) {
             ROOT::TMetaUtils::Error(0, "Module '%s' failed to load.\n",
                                     DepMod.data());

core/foundation/test/FoundationUtilsTests.cxx

@@ -22,6 +22,7 @@ using namespace ROOT::FoundationUtils;
 
 TEST(FoundationUtilsTests, CanConvertEnvValueToBool)
 {
+  ASSERT_FALSE(CanConvertEnvValueToBool(""));
   ASSERT_TRUE(CanConvertEnvValueToBool("0"));
   ASSERT_TRUE(CanConvertEnvValueToBool("false"));
   ASSERT_TRUE(CanConvertEnvValueToBool("False"));

core/metacling/src/TCling.cxx

@@ -1041,6 +1041,7 @@ static bool LoadModule(const std::string &ModuleName, cling::Interpreter &interp
       ::Info("TCling::__LoadModule", "Preloading module %s. \n",
              ModuleName.c_str());
 
+   cling::Interpreter::PushTransactionRAII deserRAII(&interp);
    return interp.loadModule(ModuleName, /*Complain=*/true);
 }
 
@@ -1115,23 +1116,23 @@ static GlobalModuleIndex *loadGlobalModuleIndex(SourceLocation TriggerLoc, cling
                TraverseDecl(TU);
             }
             bool VisitNamedDecl(NamedDecl *ND) {
-               for (auto R : ND->redecls()) {
-                  if (!R->isFromASTFile())
-                     continue;
-                  if (TagDecl *TD = llvm::dyn_cast<TagDecl>(R)) {
-                     if (TD->isCompleteDefinition())
-                        Register(TD);
-                  } else if (NamespaceDecl *NSD = llvm::dyn_cast<NamespaceDecl>(R))
-                     Register(NSD, /*AddSingleEntry=*/ false);
-                  else if (TypedefNameDecl *TND = dyn_cast<TypedefNameDecl>(R))
-                     Register(TND);
-                  // FIXME: Add the rest...
+               if (!ND->isFromASTFile())
+                  return true;
+
+               if (TagDecl *TD = llvm::dyn_cast<TagDecl>(ND)) {
+                  if (TD->isCompleteDefinition())
+                     Register(TD);
+               } else if (NamespaceDecl *NSD = llvm::dyn_cast<NamespaceDecl>(ND)) {
+                  Register(NSD, /*AddSingleEntry=*/ false);
                }
+               else if (TypedefNameDecl *TND = dyn_cast<TypedefNameDecl>(ND))
+                  Register(TND);
+               // FIXME: Add the rest...
                return true; // continue decending
             }
          private:
             clang::GlobalModuleIndex::UserDefinedInterestingIDs &DefinitionIDs;
-            void Register(NamedDecl* ND, bool AddSingleEntry = true) {
+            void Register(const NamedDecl* ND, bool AddSingleEntry = true) {
                assert(ND->isFromASTFile());
                // FIXME: All decls should have an owning module once rootcling
                // updates its generated decls from within the LookupHelper & co.
@@ -1220,18 +1221,32 @@ static void RegisterCxxModules(cling::Interpreter &clingInterp)
       clang::CompilerInstance &CI = *clingInterp.getCI();
       GlobalModuleIndex *GlobalIndex = nullptr;
       // Conservatively enable platform by platform.
-      bool supportedPlatform = false;
-      // Allow forcefully enabling the GMI.
-      llvm::Optional<std::string> EnvEnGMI = llvm::sys::Process::GetEnv("ROOT_EXPERIMENTAL_GMI");
-      if (EnvEnGMI.hasValue() && ROOT::FoundationUtils::ConvertEnvValueToBool(*EnvEnGMI))
-         supportedPlatform = true;
-
-      llvm::Optional<std::string> EnvDisGMI = llvm::sys::Process::GetEnv("ROOT_DISABLE_GMI");
-      if (EnvDisGMI.hasValue() && EnvEnGMI.hasValue())
-         ::Error("TCling__RegisterCxxModules",
-                 "Both ROOT_EXPERIMENTAL_GMI and ROOT_DISABLE_GMI env vars are set!");
-
-      if (supportedPlatform && EnvDisGMI.hasValue() && ROOT::FoundationUtils::ConvertEnvValueToBool(*EnvDisGMI)) {
+      bool supportedPlatform =
+#ifdef R__LINUX
+         true
+#elif defined(R__MACOSX)
+         true
+#else // Windows
+         false
+#endif
+         ;
+      // Allow forcefully enabling/disabling the GMI.
+      llvm::Optional<std::string> envUseGMI = llvm::sys::Process::GetEnv("ROOT_USE_GMI");
+      if (envUseGMI.hasValue()) {
+         if (!envUseGMI->empty() && !ROOT::FoundationUtils::CanConvertEnvValueToBool(*envUseGMI))
+            ::Warning("TCling__RegisterCxxModules",
+                      "Cannot convert '%s' to bool, setting to false!",
+                      envUseGMI->c_str());
+
+         bool value = envUseGMI->empty() || ROOT::FoundationUtils::ConvertEnvValueToBool(*envUseGMI);
+
+         if (supportedPlatform == value)
+            ::Warning("TCling__RegisterCxxModules", "Global module index is%sused already!",
+                     (value) ? " " :" not ");
+         supportedPlatform = value;
+      }
+
+      if (supportedPlatform) {
          loadGlobalModuleIndex(SourceLocation(), clingInterp);
          // FIXME: The ASTReader still calls loadGlobalIndex and loads the file
          // We should investigate how to suppress it completely.

core/metacling/src/TClingCallbacks.cxx

@@ -11,6 +11,8 @@
 
 #include "TClingCallbacks.h"
 
+#include <ROOT/FoundationUtils.hxx>
+
 #include "cling/Interpreter/DynamicLibraryManager.h"
 #include "cling/Interpreter/Interpreter.h"
 #include "cling/Interpreter/InterpreterCallbacks.h"
@@ -38,6 +40,7 @@
 #include "llvm/Support/Error.h"
 #include "llvm/Support/FileSystem.h"
 #include "llvm/Support/Path.h"
+#include "llvm/Support/Process.h"
 
 #include "TClingUtils.h"
 #include "ClingRAII.h"
@@ -288,8 +291,10 @@ bool TClingCallbacks::LookupObject(LookupResult &R, Scope *S) {
 
 bool TClingCallbacks::findInGlobalModuleIndex(DeclarationName Name, bool loadFirstMatchOnly /*=true*/)
 {
-   if (::getenv("ROOT_DISABLE_GMI"))
-      return false;
+   llvm::Optional<std::string> envUseGMI = llvm::sys::Process::GetEnv("ROOT_USE_GMI");
+   if (envUseGMI.hasValue())
+      if (!envUseGMI->empty() && !ROOT::FoundationUtils::ConvertEnvValueToBool(*envUseGMI))
+         return false;
 
    const CompilerInstance *CI = m_Interpreter->getCI();
    const LangOptions &LangOpts = CI->getPreprocessor().getLangOpts();
@@ -315,15 +320,25 @@ bool TClingCallbacks::findInGlobalModuleIndex(DeclarationName Name, bool loadFir
    // Find the modules that reference the identifier.
    // Note that this only finds top-level modules.
    if (Index->lookupIdentifier(Name.getAsString(), FoundModules)) {
-      for (auto FileName : FoundModules) {
-         StringRef ModuleName = llvm::sys::path::stem(*FileName);
+      for (llvm::StringRef FileName : FoundModules) {
+         StringRef ModuleName = llvm::sys::path::stem(FileName);
+
+         // Skip to the first not-yet-loaded module.
+         if (m_LoadedModuleFiles.count(FileName)) {
+            if (gDebug > 2)
+               llvm::errs() << "Module '" << ModuleName << "' already loaded"
+                            << " for '" << Name.getAsString() << "'\n";
+            continue;
+         }
+
          fIsLoadingModule = true;
          if (gDebug > 2)
-            llvm::errs() << "Loading " << ModuleName << " on demand"
-                         << " for " << Name.getAsString() << "\n";
+            llvm::errs() << "Loading '" << ModuleName << "' on demand"
+                         << " for '" << Name.getAsString() << "'\n";
 
          m_Interpreter->loadModule(ModuleName);
          fIsLoadingModule = false;
+         m_LoadedModuleFiles[FileName] = Name;
          if (loadFirstMatchOnly)
             break;
       }

core/metacling/src/TClingCallbacks.h

@@ -49,6 +49,7 @@ class TClingCallbacks : public cling::InterpreterCallbacks {
    bool fPPChanged = false;
    bool fIsCodeGening = false;
    bool fIsLoadingModule = false;
+   llvm::DenseMap<llvm::StringRef, clang::DeclarationName> m_LoadedModuleFiles;
 
 public:
    TClingCallbacks(cling::Interpreter* interp, bool hasCodeGen);

core/metacling/src/TClingDataMemberInfo.cxx

@@ -442,6 +442,8 @@ long TClingDataMemberInfo::Property() const
    property = TClingDeclInfo::Property(property, qt);
    const clang::TagType *tt = qt->getAs<clang::TagType>();
    if (tt) {
+      // tt->getDecl() might deserialize.
+      cling::Interpreter::PushTransactionRAII RAII(fInterp);
       const clang::TagDecl *td = tt->getDecl();
       if (td->isClass()) {
          property |= kIsClass;

interpreter/cling/lib/Interpreter/Interpreter.cpp

@@ -883,7 +883,6 @@ namespace cling {
     Preprocessor& PP = getCI()->getPreprocessor();
     IdentifierInfo* II = PP.getIdentifierInfo(M->Name);
     SourceLocation ValidLoc = getNextAvailableLoc();
-    Interpreter::PushTransactionRAII RAII(this);
     bool success =
        !getSema().ActOnModuleImport(ValidLoc, ValidLoc,
                                     std::make_pair(II, ValidLoc)).isInvalid();

interpreter/llvm/src/tools/clang/include/clang/Serialization/GlobalModuleIndex.h

@@ -186,7 +186,7 @@ class GlobalModuleIndex {
   /// \returns true if the identifier is known to the index, false otherwise.
   bool lookupIdentifier(StringRef Name, HitSet &Hits);
 
-  typedef llvm::SmallPtrSet<std::string *, 4> FileNameHitSet;
+  typedef llvm::SmallDenseSet<llvm::StringRef, 4> FileNameHitSet;
   bool lookupIdentifier(StringRef Name, FileNameHitSet &Hits);
 
   /// \brief Note that the given module file has been loaded.

interpreter/llvm/src/tools/clang/lib/Serialization/GlobalModuleIndex.cpp

@@ -339,7 +339,7 @@ bool GlobalModuleIndex::lookupIdentifier(StringRef Name, FileNameHitSet &Hits) {
   SmallVector<unsigned, 2> ModuleIDs = *Known;
   for (unsigned I = 0, N = ModuleIDs.size(); I != N; ++I) {
     assert(!Modules[ModuleIDs[I]].FileName.empty());
-    Hits.insert(&Modules[ModuleIDs[I]].FileName);
+    Hits.insert(Modules[ModuleIDs[I]].FileName);
   }
 
   ++NumIdentifierLookupHits;