Request _C++20_ module compatibility with ROOT 6.30/00

[greenc-FNAL]

Feature description

As C++20 goes mainstream, there are many reasons to start using C++20 modules in a non-ROOT context to provide well-designed and encapsulated APIs. Generated module files are compiler-specific, which would seem likely to lead to issues with interactions between Cling-generated PCM dictionaries and any C++20 modules used by the code upon which those dictionaries depend.

My request is for the upcoming ROOT with Clang 16 to be able to accommodate the generation and use of PCM dictionaries where dependencies thereof are or use C++20 modules. This may or may not involve transitioning PCM dictionary generation to use C++20 modules as opposed to Clang's original implementation of C++ modules.

Alternatives considered

Short of having to operate under the constraint that no code depended upon by a ROOT dictionary be or use C++20 modules at any level (direct or indirect), I do not see an alternative to providing the requested feature, although it is certainly possible I am overestimating the complexity of the situation and that the feature request either is already implemented or, "just works" already in 6.28/04.

Additional context

In dealing with #13461 it was suggested by @pcanal and later verified in vitro that using PCM dictionaries vs .rootmap dictionaries would be a viable workaround. This led to thoughts about ROOT C++ modules vs non-ROOT C++20 module generation, and eventually this feature request.

[vgvassilev]

Hi @greenc-FNAL,

Thanks for this request. I will try to reply inline directly to some of the concerns and try to give some public-friendly outline the bigger picture as I see it.

Generated module files are compiler-specific, which would seem likely to lead to issues with interactions between Cling-generated PCM dictionaries and any C++20 modules used by the code upon which those dictionaries depend.

I am not sure if that is the case at least for the next 10 years. I see this can be an issue if we start shipping module files (bmis or equivalent) instead of header files. However, in practice that won't be the case in near future because the compiler vendors cannot agree on a common file format and make it standard. Until then, there will be always some sort of a textual header file which can be processed with our clang-based rootcling and build a pcm file as basis for our dictionaries.

My request is for the upcoming ROOT with Clang 16 to be able to accommodate the generation and use of PCM dictionaries where dependencies thereof are or use C++20 modules.

Clang has at least 5 different "flavors" of modules. One of them is the C++20 modules as described by the C++ standard. At the moment you start using C++ 20 you could use the import/export constructs and you will be able to use that feature for encapsulation purposes. There is not enough project transitions which are bigger than toy examples. One of the challenges is how we discover module dependencies. That is, we need extra tooling to decide how to split the source files into modules. In practice we need to pre-lex all of the content before the build system can start processing project files. The community is discussing scanners (such as clang scan-deps), daemons (the gcc implementation), protocols, etc. There is increasing amount of papers trying to address how we put C++ modules at scale (https://wg21.link/p2656, https://wg21.link/P2409, https://youtu.be/_LGR0U5Opdg?si=AbLazREvyl5PXVFG). Some papers/talks are more radical claiming that we cannot even roll out modules (as written in the standard) at such scale.

For modules to be useful and work reliably we know that we need to apply modularization bottom up. That means that before modularizing cmssw/art we need everything below starting from libc and libxml to become modular. That's something we observed during our modularization efforts in cmssw (see cms-sw/cmssw#15248). The reasons we cannot make it yet default is discussed here: cms-sw/cmssw#41810 (comment) and here #13000 Not surprisingly we have observed similar comments coming from modularization efforts in gcc and msvc.

Let's assume we live in an platonic world where all of the above is resolved. It still seems we will be able to use our dictionary system as it is because we will in the end have a build system that processes textual files to build them. There we could use any custom flavor of modules (or something better) that makes our I/O work. We can go one step further and claim that, even if we introduce strict interface encapsulation via C++ 20 modules, our I/O system would require to know more about the non-exported entities so that we can serialize and deserialize the private dependencies of the exported entities. That is, we will probably end up with a very similar system that we have today.

I would personally like to tackle the engineering challenge as you have described. However I believe we are not there yet and any efforts into trying to adapt to a still volatile area would probably cause disturbances in our user base for not yet clear final goal.

Does that address/answer the concerns you had when you opened that request?

[greenc-FNAL]

@vgvassilev Thanks for your detailed and thoughtful reply. I'm confused about one thing in particular though: in the light of https://en.cppreference.com/w/cpp/language/modules#Global_module_fragment (as pointed out to me by my colleague @knoepfel), it would not seem to be a given in the general case that bottom-up modularization is required (or even desired). I think I understand your argument re serialization, but I'm not sure how that impacts our request to have ROOT support interacting with externally-generated C++20 modules.

I will look at the materials you reference and attempt to do some further reading on the subject. It does seem though, that there's something of a chicken-and-egg problem here: unless ROOT is proactive in developing support for external use of C++20 modules, then it becomes difficult for ROOT-using projects to even consider using C++20 modules in their own projects if it is likely to intersect with their use of ROOT.

[vgvassilev]

@vgvassilev Thanks for your detailed and thoughtful reply. I'm confused about one thing in particular though: in the light of https://en.cppreference.com/w/cpp/language/modules#Global_module_fragment (as pointed out to me by my colleague @knoepfel), it would not seem to be a given in the general case that bottom-up modularization is required (or even desired). I think I understand your argument re serialization, but I'm not sure how that impacts our request to have ROOT support interacting with externally-generated C++20 modules.

That is correct since that is implementation/vendor-specific. You can follow the discussions on the SG15 side at isocpp.

I will look at the materials you reference and attempt to do some further reading on the subject. It does seem though, that there's something of a chicken-and-egg problem here: unless ROOT is proactive in developing support for external use of C++20 modules, then it becomes difficult for ROOT-using projects to even consider using C++20 modules in their own projects if it is likely to intersect with their use of ROOT.

The C++ modules support in the dictionaries is based on "Clang modules". This is the very same implementation of the "C++ 20 modules". In the end the implementation in Clang has driven the standardization of the functionality. In practice, modulo the import/export syntax and a few quirks what ROOT uses is the basis of what you call here "C++20 modules".

If you provide a concrete, close-to-production example that works across various vendors (clang and gcc, lets exclude for now msvc) where ROOT is the blocker I can take a look at it and figure out how to support it.

[greenc-FNAL]

Thanks again for the thoughtful reply.

If you provide a concrete, close-to-production example that works across various vendors (clang and gcc, lets exclude for now msvc) where ROOT is the blocker I can take a look at it and figure out how to support it.

I would like to do this; the question I have is, against what ROOT version/branch would it be most useful to work? Is 6.28/04 sufficient with LLVM/Clang 13, or should I be using master or something else?

[vgvassilev]

Either branch should be sufficient. I'd prefer the master.

[dpiparo]

Is this issue to be considered closed for 6.32, @vgvassilev ?

[vgvassilev]

I do not think the outcome of the discussion was sufficient to make @greenc-FNAL happy. Although we should move their codes to use modules in dictionaries first …