Please consider installing std.cppm in lib++

[boris-kolpackov]

I've been trying out the std module in libc++ with build2 (per the Modules in libc++ documentation) and everything seems to work without any issues, at least on simple examples. Very impressive, I must say.

There is just one snag: std.cppm is not installed. Quoting the relevant part from the above documentation:

Once libc++'s implementation is more mature we will reach out to build system vendors, with the goal that building the BMI files is done by the build system.

Currently this requires a local build of libc++ with modules enabled. Since modules are not part of the installation yet, they are used from the build directory.

In build2 we have the ability to build BMIs of external libraries on the fly (this is needed for consuming modules from installed libraries). So if std.cppm were installed somewhere where we could find it, then this would work pretty much out of the box in our case.

I can see two immediate questions about installing std.cppm: where to install it and how to make it self-contained (currently, it #includes a large number of files from the std/ subdirectory).

Regarding where to install, I would suggest just installing it next to the headers (e.g., into /usr/include/c++/v1/). I can think of two advantage with doing it this way:

1. There is already a way for a build system to find this directory (by parsing the header search paths in the -v output) and some build systems already do this (for example, in build2 we extract the compiler's header and library search paths).

2. The std.cppm file will end up in the right distribution package (e.g., -dev on Debian, -devel on Fedora, etc) without any extra effort.

The only objection I've heard to installing modules next to headers is that "it's not a header" and "it may be included by mistake". While both are true, we are already installing non-header files (e.g., inline/template implementation files) and nobody seems to be including them by mistake. We could also take an extra measure to preclude this by, for example, installing the modules into a subdirectory (e.g., /usr/include/c++/v1/modules/).

Regarding making std.cppm self-contained, I assume there is no desire to install all the 100+ files from std/ that it includes (though there is nothing technically bad about doing it, just feels unnecessary). It seems the easiest would be to just append their contents to std.cppm.in instead of #includeing them. Though I am not sure if this is easy to achieve in CMake.

Thoughts?

P.S. For completeness, let me also mention what I've done as a (hopefully) temporary measure in build2 to support import std right now. I've made a self-contained std.cppm (as described above), patched it slightly so that it works for both libc++ 17 and 18, and bundled it with build2. In other words, instead of being installed with libc++, it is being installed with build2 (again, hopefully temporarily and into build2's own directory, naturally).

@mordante @iains @ChuanqiXu9 @dwblaikie @mathstuf

[llvmbot]

@llvm/issue-subscribers-clang-modules

Author: Boris Kolpackov (boris-kolpackov)

I've been trying out the `std` module in `libc++` with `build2` (per the [Modules in libc++](https://github.com/llvm/llvm-project/blob/main/libcxx/docs/Modules.rst) documentation) and everything seems to work without any issues, at least on simple examples. Very impressive, I must say.

There is just one snag: std.cppm is not installed. Quoting the relevant part from the above documentation:

> Once libc++'s implementation is more mature we will reach out to build system vendors, with the goal that building the BMI files is done by the build system.
>
> Currently this requires a local build of libc++ with modules enabled. Since modules are not part of the installation yet, they are used from the build directory.

In build2 we have the ability to build BMIs of external libraries on the fly (this is needed for consuming modules from installed libraries). So if std.cppm were installed somewhere where we could find it, then this would work pretty much out of the box in our case.

I can see two immediate questions about installing std.cppm: where to install it and how to make it self-contained (currently, it #includes a large number of files from the std/ subdirectory).

Regarding where to install, I would suggest just installing it next to the headers (e.g., into /usr/include/c++/v1/). I can think of two advantage with doing it this way:

1. There is already a way for a build system to find this directory (by parsing the header search paths in the -v output) and some build systems already do this (for example, in build2 we extract the compiler's header and library search paths).

2. The std.cppm file will end up in the right distribution package (e.g., -dev on Debian, -devel on Fedora, etc) without any extra effort.

The only objection I've heard to installing modules next to headers is that "it's not a header" and "it may be included by mistake". While both are true, we are already installing non-header files (e.g., inline/template implementation files) and nobody seems to be including them by mistake. We could also take an extra measure to preclude this by, for example, installing the modules into a subdirectory (e.g., /usr/include/c++/v1/modules/).

Regarding making std.cppm self-contained, I assume there is no desire to install all the 100+ files from std/ that it includes (though there is nothing technically bad about doing it, just feels unnecessary). It seems the easiest would be to just append their contents to std.cppm.in instead of #includeing them. Though I am not sure if this is easy to achieve in CMake.

Thoughts?

P.S. For completeness, let me also mention what I've done as a (hopefully) temporary measure in build2 to support import std right now. I've made a self-contained std.cppm (as described above), patched it slightly so that it works for both libc++ 17 and 18, and bundled it with build2. In other words, instead of being installed with libc++, it is being installed with build2 (again, hopefully temporarily and into build2's own directory, naturally).

@mordante @iains @ChuanqiXu9 @dwblaikie @mathstuf

[ChuanqiXu9]

FYI, about the position of installing modules, there is a discussion in https://lists.isocpp.org/sg15/2023/07/1971.php

[boris-kolpackov]

BTW, to add a data point for the "perhaps the std module in libc++ is mature enough to start installing it" position: I was able to build @ChuanqiXu9 's simple_async CXX20Modules branch with libc++. All I had to do is fix a few bugs in the project and get rid of one workaround (which is not standard C++). You can see the changes here: https://github.com/boris-kolpackov/async_simple/tree/CXX20Modules-build2

[mathstuf]

Note that we don't just need the files, but also the flags to make them work as expected (namely to suppress warnings about reserved module names via -Wno-reserved-module-identifier).

[ldionne]

If we start shipping std.cppm at a given location, that will become the de-facto standard for build systems to pick up modules files from. I'm not really opposed to doing that but it seems like we'd be stepping on SG15's toes since my understanding is that we haven't come to an official agreement on that yet?

CC @Bigcheese

[mordante]

Great to hear it works with build2!

Shipping the module files is getting at the top of my todo list and I wanted to investigate which tool vendors to contact. However last week I saw @ruoso created cplusplus/modules-ecosystem-tr#29. So I prefer to participate in that effort instead of starting a similar effort. Build2 is also on that list.

Regarding the large amount of inc files. For now I intend to keep them, they are very useful for testing that libc++ exports all named declarations from a header.

[boris-kolpackov]

Note that we don't just need the files, but also the flags to make them work as expected (namely to suppress warnings about reserved module names via -Wno-reserved-module-identifier).

FWIW, this is the only extra option I had to pass to compile std.cppm. At least in build2 this whole logic is quite ad hoc: detect the importation of the std modules, notice that it's clang with libc++, find std.cppm, synthesize the dependency for the BMI, and add the extra option. And I can't see any major generalization in this area. Feels like build systems will just need to handle each compiler/standard library combination. Luckily there aren't that many of them.

I'm not really opposed to doing that but it seems like we'd be stepping on SG15's toes since my understanding is that we haven't come to an official agreement on that yet?

How can I put this politely? SG15 still hasn't come to an official agreement on the file extension for module but it seems to have agreed on the term used for compiled modules: BMI. Though there is no agreement on what it stands for: some think it's a "binary module interface", some think it's a "built module interface", there is probably another interpretation I am not aware of. So if you wait on SG15, you may end up in a situation where build systems just run out of patience and start bundling your module interfaces.

Also, FWIW, MSVC folks have decided on the location and are guaranteeing it: build2/build2#333 (comment)

Regarding the large amount of inc files. For now I intend to keep them, they are very useful for testing that libc++ exports all named declarations from a header.

Just to clarify, I didn't suggest that you get rid of them entirely, only in the installed version of std.cppm. Even that was only a suggestions -- if you decide to install them, I don't think anyone will complain.

[mathstuf]

Feels like build systems will just need to handle each compiler/standard library combination. Luckily there aren't that many of them.

There are at least a half dozen clang-based frontends with different options and no telling module flags will be one such disagreement point or not (hopefully, but some have different flag patterns they may wish to follow). It don't think it is so trivial unless one only wants to support upstream clang, Apple clang, gcc, and MSVC.

[ChuanqiXu9]

From cplusplus/modules-ecosystem-tr#29:

The first step of this process will be to collect input from a wide range of tooling providers in order to identify requirements in the ecosystem, for that we need to identify who we need to reach out to in order to get the requirements.

So it looks like that issue is not a good place to talk about the installation place for std modules.

If we want to push things forward, maybe we can send another mail to SG15 with a stronger desire to get a conclusion (instead of purely discussion), or if we want, we can make the decision ourselves. (I don't have a strong opinion here)

CC: @mordante @ldionne @boris-kolpackov

[Bigcheese]

The specific search paths used to find module source files in general hasn't been something that SG15 has discussed so far. This is largely because that very quickly hits packaging, project layout, and distribution, which are things we care about, but are large issues that would delay getting the minimum needed for modules.

I do think that discussing the limited case of where libc++ and libstdc++ should put their std module is something that's useful to discuss in SG15, and I'd be happy to setup a meeting soon to discuss it.

[boris-kolpackov]

If we want to push things forward, maybe we can send another mail to SG15 with a stronger desire to get a conclusion (instead of purely discussion), or if we want, we can make the decision ourselves.

I don't see an issue with making a decision ourselves. MSVC has already decided and if we pick something sensible I don't see a reason why libstdc++ folks won't adopt it.

I think in the end it doesn't really matter where the modules are installed provided that:

1. It doesn't cause friction for downstream package maintainers in Debian, Fedora, etc. For example, choosing something next to /usr/include/ such as /usr/modules/ will definitely cause friction.

2. It's reasonably easy for a build system to determine this location. Paths printed with -print search-dirs would be one natural place. Alternatively, something like -print-file-name=modules by analogy with -print-file-name=plugin in GCC.

The reason I personally like /usr/include/ is because it's a non-event for downstream maintainers and any serious build system already has to extract header search paths (for example, to suppress duplicates in pkg-config options, etc). But I am ok with anything else sensible.

[mathstuf]

I prefer (underneath) /usr/lib because I predict that we'll have more arch-specific files than we already have with headers necessitating some arch-ful bits in /usr/include.

[ldionne]

I encourage you (@mathstuf and @boris-kolpackov) to take part in the SG15 discussion. Let's cut a deal here. We all want to make progress on shipping modules, and I think we would all like somewhat consistent paths across implementations to keep things simple (at least I and @Bigcheese do).

Let's schedule a SG15 discussion, have it, and if there is no conclusion at the end of this SG15 discussion, then we'll do whatever we want while consulting with @jwakely to align with libstdc++. Let's at least try getting agreement within SG15 and if this doesn't seem to work, then we'll just unblock our work because I agree we need to get the ball rolling on this if we want users to start using it.

[boris-kolpackov]

I predict that we'll have more arch-specific files than we already have with headers

Do any plausible examples come to mind?

necessitating some arch-ful bits in /usr/include.

There is already a convention for installing arch-specific headers into /usr/include/<target-triplet>/, for example into /usr/include/x86_64-linux-gnu/ on my machine. We've even added an installation location name for this recently: https://github.com/build2/build2/blob/v0.16.0/NEWS#L257-L272

[jwakely]

There is already a convention for installing arch-specific headers into /usr/include/<target-triplet>/, for example into /usr/include/x86_64-linux-gnu/ on my machine.

That's specific to Debian-based distros that follow Debian's MultiArch convention. That's not universal.

[mathstuf]

Do any plausible examples come to mind?

Anything that does SIMD will have per-arch modules unless the backend is 100% isolated (which seems very unlikely to me) will expose transitive arch-specific modules. Maybe it's resolvable with preprocessor arch detection and things Just Work on Linux? Not sure how this works for macOS universal builds though.

[ChuanqiXu9]

The specific search paths used to find module source files in general hasn't been something that SG15 has discussed so far. This is largely because that very quickly hits packaging, project layout, and distribution, which are things we care about, but are large issues that would delay getting the minimum needed for modules.

I do think that discussing the limited case of where libc++ and libstdc++ should put their std module is something that's useful to discuss in SG15, and I'd be happy to setup a meeting soon to discuss it.

Can we have a meeting in 2 weeks? Since Clang/LLVM18 will be branched in the end of the next month, I really hope we can achieve this before that.

[Bigcheese]

The specific search paths used to find module source files in general hasn't been something that SG15 has discussed so far. This is largely because that very quickly hits packaging, project layout, and distribution, which are things we care about, but are large issues that would delay getting the minimum needed for modules.
I do think that discussing the limited case of where libc++ and libstdc++ should put their std module is something that's useful to discuss in SG15, and I'd be happy to setup a meeting soon to discuss it.

Can we have a meeting in 2 weeks? Since Clang/LLVM18 will be branched in the end of the next month, I really hope we can achieve this before that.

I'll send out a scheduling poll to figure out when the best time to meet is either next week or the week after. It would also be good to have some more detailed info for the other suggestions in the thread (what Boris wrote is a good level of detail for a discussion).

[fweimer-rh]

The error towards the end of https://github.com/llvm/llvm-project/blob/main/libcxx/docs/Modules.rst suggests that std.cppm itself is not portable across different compiler flags:

error: module file _deps/std-build/CMakeFiles/std.dir/std.pcm cannot be loaded due to a configuration mismatch with the current compilation [-Wmodule-file-config-mismatch]

If that's really true (and not just for compiled module representation), it's not a good idea to ship it. Before we can do that, we need something that can be used to produce a compiled module for -fexceptions, -fno-exceptions, different C++ revisions, and so on.

[Bigcheese]

The error towards the end of https://github.com/llvm/llvm-project/blob/main/libcxx/docs/Modules.rst suggests that std.cppm itself is not portable across different compiler flags:

error: module file _deps/std-build/CMakeFiles/std.dir/std.pcm cannot be loaded due to a configuration mismatch with the current compilation [-Wmodule-file-config-mismatch]

If that's really true (and not just for compiled module representation), it's not a good idea to ship it. Before we can do that, we need something that can be used to produce a compiled module for -fexceptions, -fno-exceptions, different C++ revisions, and so on.

That's for the produced BMI file, which is a binary file that is the actual compiled module interface. std.cppm is the source file from which the BMI is produced.

[mordante]

The error towards the end of https://github.com/llvm/llvm-project/blob/main/libcxx/docs/Modules.rst suggests that std.cppm itself is not portable across different compiler flags:

error: module file _deps/std-build/CMakeFiles/std.dir/std.pcm cannot be loaded due to a configuration mismatch with the current compilation [-Wmodule-file-config-mismatch]

If that's really true (and not just for compiled module representation), it's not a good idea to ship it. Before we can do that, we need something that can be used to produce a compiled module for -fexceptions, -fno-exceptions, different C++ revisions, and so on.

This is why modules need build system support, they know the user's compilation flags. Then they can create the BMI(s) that match the compiler flags. As @Bigcheese mentioned the source std.cppm is portable.

[fweimer-rh]

I see. Do we need to materialize the source file on disk? Would it be sufficient if the compiler driver had an option to build the std binary module representation according to the other compilation flags?

[boris-kolpackov]

Would it be sufficient if the compiler driver had an option to build the std binary module representation according to the other compilation flags?

You would then have to provide a mechanism for detecting when the previously compiled std module BMI is out-of-date. Also, presumably, at some point there will be the std module in libstdc++ which we would want to support. Feels like installing the std module source and letting the build system take care of the rest is the most sensible approach.

[fweimer-rh]

Would it be sufficient if the compiler driver had an option to build the std binary module representation according to the other compilation flags?

You would then have to provide a mechanism for detecting when the previously compiled std module BMI is out-of-date.

Dependency tracking options would already provide that kind of information? Just looking at the file modification time won't cover dependent files even if the source module is materialized on disk.

Also, presumably, at some point there will be the std module in libstdc++ which we would want to support. Feels like installing the std module source and letting the build system take care of the rest is the most sensible approach.

That probably needs a more complicated handshake between the projects anyway, and a single source file might not deliver all the required ingredients.

[boris-kolpackov]

Let me copy what I believe is the relevant reply from the SG15 mailing list:

In my mind, the motivation of the meeting should be the location of the std modules. Should it be <PREFIX>/modules/c++/v1? Should it be /usr/include/c++/$version? Or should it be $PREFIX/share/c++/modules?

The consensus is that it should be intermediated by the metadata file, which makes the specific location less important, and open for package managers to customize.

There was also a brief conversation where the feeling was that we shouldn't reuse the include directory for this, and instead this should be inside a private location for the standard library.

So my summary is SG15 added another level of indirection (the metadata file), but where to install it or the actual module interface units as well as how to convey this information to build systems is left as an exercise to the implementers.

[iains]

Let me copy what I believe is the relevant reply from the SG15 mailing list:

In my mind, the motivation of the meeting should be the location of the std modules. Should it be <PREFIX>/modules/c++/v1? Should it be /usr/include/c++/$version? Or should it be $PREFIX/share/c++/modules?

The consensus is that it should be intermediated by the metadata file, which makes the specific location less important, and open for package managers to customize.
There was also a brief conversation where the feeling was that we shouldn't reuse the include directory for this, and instead this should be inside a private location for the standard library.

So my summary is SG15 added another level of indirection (the metadata file), but where to install it or the actual module interface units as well as how to convey this information to build systems is left as an exercise to the implementers.

Another way of looking at this is that SG15 is trying to determine a mechanism (and actually to specify the content of the metadata file) but giving maximum freedom to the toolchain and OS implementations in how they wish to lay things out.

(my take on) The underlying theme is

• that the toolchain knows how to find libraries (whether that is done by the driver or some other sub-tool)
• the relevant metadata files need to be tied to a library ABI (in the case of toolchains supporting multilibs, for example). So that the toolchain is already making that distinction (in order to link the correct ABI library) and should be able to make the same distinction for the metadata.

perhaps a reasonable initial implementation would be to place the metadata alongside each library ABI slice (and then the same driver/sub-tool process can be used to locate it based on ABI-changing flags [e.g. hard/soft float or 32 / 64B ISA]).

Please make input on the SG15 reflector in helping figure out the initial metadata content, and in comments/improvements to the scheme in general (it must still be considered Work In Progress).

[boris-kolpackov]

Another way of looking at this is that SG15 is trying to determine a mechanism (and actually to specify the content of the metadata file) but giving maximum freedom to the toolchain and OS implementations in how they wish to lay things out.

From the build system's POV (which is the primary consumer of this mechanism), it doesn't make a difference whether the compiler gives it the location of the standard modules directly or it give it the location of the metadata file from which the build system gets those locations. If anything, it's an extra hoop to jump through. I do understand that this metadata file may also provide extra information (compile options, etc) that is necessary to build the modules, but it is not the case currently. More precisely, nothing extra is needed for compiling MS STL modules and for libc++ only -Wno-reserved-module-identifier is needed, which, at least in case of build2, is a lot easier to add ad hoc than to parse a JSON file. So from my POV, SG15 is focusing on theoretical problems that we currently don't have (but may have in the future, admittedly) while ignoring practical problems that we do have (like helping implementers decide where to install module interfaces and how to communicate this to the build systems).

However,

This issue has been discussed in SG-15 and progress was made. So I expect we can have something before branching the next release (the end of January 2024).

this is all that really matters. If the SG15 discussion helped and we can start using import std; in Clang 18, I am happy to hear that. I can parse an extra JSON file or two.

[Bigcheese]

So my summary is SG15 added another level of indirection (the metadata file), but where to install it or the actual module interface units as well as how to convey this information to build systems is left as an exercise to the implementers.

The metadata file is needed because the flags needed to build the module interface for the std module cannot be determined by looking at the input command line unless you want to hard code that information in every build system. For example you need the extra flag to say that you're allowed to name a module std, and for the MS stdlib you need to say don't treat anything as header units, even if the importer is.

Additionally for ABI breaking differences, the options you need are directly tied to the library binary being used, not to the source code the interface was built from.

Clang and GCC currently plan to use the already existing -print-file-name to provide the path to this file to the build system.

[ChuanqiXu9]

While I respect the work of SG15, will it be a blocking issue for shipping std modules in libc++?

I mean, for example, can we use the proposal from Daniel as the first step?

{
  "version": 1,
  "revision": 1,
  "modules": [
    {
      "logical-name": "std",
      "source-path": "modules/std.cppm",
      "is-standard-library": true
    },
    {
      "logical-name": "std.compat",
      "source-path": "modules/std.compat.cppm"
      "is-std-library": true
    },
  ]
}

It doesn't solve the issue you raised but it allows us to ship module units initially so that people can start experimenting and get an initial feeling. Also I think it won't add extra works in this direction (add flags to the metadata file).

[fweimer-rh]

The metadata file is needed because the flags needed to build the module interface for the std module cannot be determined by looking at the input command line unless you want to hard code that information in every build system. For example you need the extra flag to say that you're allowed to name a module std, and for the MS stdlib you need to say don't treat anything as header units, even if the importer is.

I still don't see why you couldn't use a simple flag that tells the compiler to write out the binary object representation matching the rest of the command line flags. The build system already needs compiler-specific knowledge how to set the output path, how to request compilation instead of linking, and so on, so knowledge of this new flag seems a rather minor addition. In this approach, is up to the compiler to get itself into a state were it is able to create a module named std, so a whole bunch of potential issues simply do not apply.

[mordante]

While I respect the work of SG15, will it be a blocking issue for shipping std modules in libc++?

Quite the opposite, the discussion in SG15 gave a good direction for a way forward.

I mean, for example, can we use the proposal from Daniel as the first step?

Yes. I plan to look at that soon so build systems can use it.

The metadata file is needed because the flags needed to build the module interface for the std module cannot be determined by looking at the input command line unless you want to hard code that information in every build system. For example you need the extra flag to say that you're allowed to name a module std, and for the MS stdlib you need to say don't treat anything as header units, even if the importer is.

I still don't see why you couldn't use a simple flag that tells the compiler to write out the binary object representation matching the rest of the command line flags. The build system already needs compiler-specific knowledge how to set the output path, how to request compilation instead of linking, and so on, so knowledge of this new flag seems a rather minor addition. In this approach, is up to the compiler to get itself into a state were it is able to create a module named std, so a whole bunch of potential issues simply do not apply.

It would require the compiler to know things it currently doesn't know. For example when the vendor builds libc++ without support for exceptions the compiler needs to issue a -fno-exception flag when building the modules. This is not known when the compiler is being built. Having a separate file with the meta data reduces the coupling between the compiler and the library.

[boris-kolpackov]

I mean, for example, can we use the proposal from Daniel as the first step?

FWIW, this metadata file would be sufficient for my needs. I have no problem hardcoding -Wno-reserved-module-identifier, which is the least of my problems in this area.

For example when the vendor builds libc++ without support for exceptions the compiler needs to issue a -fno-exception flag when building the modules.

This is actually an interesting question. Let's say libc++ was compiled with -fno-exceptions while an application that tries to use it is not. Presumable this should result in an error at some point or maybe not? If there should be an error, I can see two way it can play out:

1. The build system compiles std with -fno-exceptions (from the metadata file). Then when this BMI is used to compile an application's translation unit, there is presumably an error saying the BMI was built with incompatible options.

2. The build system compiles std with flags of the application (and thus without -fno-exceptions). Do we expect this to succeed? If so, will there be an error at some later stage (seems like it can only happen during linking)?

From the user's perspective, it seems the ideal outcome would be to fail when building the std module BMI. Not sure if it's practically doable though. Maybe this was all covered in the SG15 discussion, in which case apologies for the noise.

[iains]

I mean, for example, can we use the proposal from Daniel as the first step?
and that is related to / takes into account the existing practice for MSVC.

For example when the vendor builds libc++ without support for exceptions the compiler needs to issue a -fno-exception flag when building the modules.

This is actually an interesting question. Let's say libc++ was compiled with -fno-exceptions while an application that tries to use it is not. Presumable this should result in an error at some point or maybe not? I

I think that depends on whether there are two instances of the library binary, one with and one without exceptions. If that is the case, then the user's flags would be used to select which one (I'd assume) and therefore there should be two metadata files one for each incompatible library build.

If there is only one build (e.g. without exceptions) then, presumably the -fno-exceptions flags should be added from the metadata (i.e. as a 'required build flag'). An alternate is that the library headers emit a diagnostic in the case that the user's flags conflict [since the presence of that flag is usually reflected in some cpp define] (i.e. we assume that the person building without exceptions did that in a deliberate way that can be detected during builds).

edit: in general I think we can assume that toolchain vendors do not set out to make their user's lives hard(er) so that some reasonable mechanism should exist for discovery of constraints (even if that is only documentation).

[boris-kolpackov]

I think that depends on whether there are two instances of the library binary, one with and one without exceptions. If that is the case, then the user's flags would be used to select which one (I'd assume) and therefore there should be two metadata files one for each incompatible library build.

Right, though I was specifically drilling down on the case where the standard library flags are incompatible with the user flags. But this suggests a third option in my list above:

3. The build system invokes the compiler with -print-file-name=std-modules but without -fno-exception, the compiler see that there is no suitable version of the standard library, and returns nothing. The build system then reports to the user that no suitable version of the standard library is available for their build configuration. I like that, if we can swing it.

[mordante]

I've created an RFC PR (#75741) for the libc++ changes.

[boris-kolpackov]

@mordante Should this issues be reopened seeing that the commit in question has been reverted?

Also, I see that Clang 18 rc1 has been published which makes me wonder if the ship has sailed on having this functionality in 18. Or are you still planning/hoping to get it in before the final release?

[ChuanqiXu9]

@mordante Should this issues be reopened seeing that the commit in question has been reverted?

I don't think we should reopen this since the revered patch is in the clang part. And the job of the libc++ part, to provide option to install it, is completed.

[boris-kolpackov]

I don't think we should reopen this since the revered patch is in the clang part. And the job of the libc++ part, to provide option to install it, is completed.

This feels like splitting hairs to me since there is little use of the installed libc++ modules if nobody can find them. And to me this issue was nicely tracking the overall functionality (compared to a number of RFCs and PRs that were generated to address this) that would be needed by a build system. But Ok.

[boris-kolpackov]

I am trying to understand (and document) what ended up making it into Clang 18 specifically looking at Clang 18.1.5 in Debian (the latest release at the time of writing).

The -print-library-module-manifest-path option, it appears, did not make it:

clang++-18 -std=c++23 -stdlib=libc++ -print-library-module-manifest-path
clang++-18: error: unknown argument: '-print-library-module-manifest-path'
clang++-18: error: no input files

This seems to match #85637, which is an attempt to backport this to 18 that was abandoned.

However, the JSON file and the std/std.compat source files appear to be present:

$ cat /usr/lib/llvm-18/lib/libc++.modules.json 
{
  "version": 1,
  "revision": 1,
  "modules": [
    {
      "logical-name": "std",
      "source-path": "../share/libc++/v1/std.cppm",
      "is-std-library": true,
      "local-arguments": {
        "system-include-directories": [
          "../share/libc++/v1"
        ]
      }
    },
    {
      "logical-name": "std.compat",
      "source-path": "../share/libc++/v1/std.compat.cppm",
      "is-std-library": true,
      "local-arguments": {
        "system-include-directories": [
          "../share/libc++/v1"
        ]
      }
    }
  ]
}

ls -1 /usr/lib/llvm-18/lib/../share/libc++/v1/
std/
std.compat/
std.compat.cppm
std.cppm

Is this understanding accurate or am I missing something? If it is accurate, is it also accurate to assume that the full support for this will only be available in Clang 19?

[ChuanqiXu9]

I am trying to understand (and document) what ended up making it into Clang 18 specifically looking at Clang 18.1.5 in Debian (the latest release at the time of writing).

The -print-library-module-manifest-path option, it appears, did not make it:

clang++-18 -std=c++23 -stdlib=libc++ -print-library-module-manifest-path
clang++-18: error: unknown argument: '-print-library-module-manifest-path'
clang++-18: error: no input files

This seems to match #85637, which is an attempt to backport this to 18 that was abandoned.

However, the JSON file and the std/std.compat source files appear to be present:

$ cat /usr/lib/llvm-18/lib/libc++.modules.json 
{
  "version": 1,
  "revision": 1,
  "modules": [
    {
      "logical-name": "std",
      "source-path": "../share/libc++/v1/std.cppm",
      "is-std-library": true,
      "local-arguments": {
        "system-include-directories": [
          "../share/libc++/v1"
        ]
      }
    },
    {
      "logical-name": "std.compat",
      "source-path": "../share/libc++/v1/std.compat.cppm",
      "is-std-library": true,
      "local-arguments": {
        "system-include-directories": [
          "../share/libc++/v1"
        ]
      }
    }
  ]
}

ls -1 /usr/lib/llvm-18/lib/../share/libc++/v1/
std/
std.compat/
std.compat.cppm
std.cppm

Is this understanding accurate or am I missing something? If it is accurate, is it also accurate to assume that the full support for this will only be available in Clang 19?

Yes, I think this is accurate. BTW, maybe it is helpful to mention that, whether or not the json file gets installed depends on vendors choices.

CC @mordante and @mathstuf for double check.

[boris-kolpackov]

Another data point: I've installed Clang 19 snapshot from Debian experimental:

Debian clang version 19.0.0 (++20240421021844+e095d978ba47-1~exp1)

In this version the -print-library-module-manifest-path option is recognized but it doesn't print anything sensible:

$ clang++-19 -std=c++23 -stdlib=libc++ -print-library-module-manifest-path
<NOT PRESENT>

Even though the JSON file and the std/std.compat source files are installed, the same as for Clang 18.

[ChuanqiXu9]

Another data point: I've installed Clang 19 snapshot from Debian experimental:

Debian clang version 19.0.0 (++20240421021844+e095d978ba47-1~exp1)

In this version the -print-library-module-manifest-path option is recognized but it doesn't print anything sensible:

$ clang++-19 -std=c++23 -stdlib=libc++ -print-library-module-manifest-path
<NOT PRESENT>

Even though the JSON file and the std/std.compat source files are installed, the same as for Clang 18.

Oh, this should be a bug IIRC. @mordante. Maybe we didn't receive bug reports just because no one uses it : (

Can you compile a hello world example with -std=libc++ in this configuration?

[mathstuf]

The -print-file-name=libc++.modules.json flag works for CMake to find the file.

[boris-kolpackov]

Can you compile a hello world example with -std=libc++ in this configuration?

Yes. I can even do one better and compile it with import std; if I connect things manually.

The -print-file-name=libc++.modules.json flag works for CMake to find the file.

Hm, yes, this works for me also with Clang 19 (but not 18).

Now I am wondering what is the difference/purpose of -print-library-module-manifest-path? Can someone give an authoritative answer as to what mechanism a build system vendor should use?

[mathstuf]

I tested it with a custom-built 18.1.2, so why 18 doesn't work in your case is an interesting question…