Please consider installing std.cppm in lib++

[boris-kolpackov]

I've been trying out the std module in libc++ with build2 (per the Modules in libc++ documentation) and everything seems to work without any issues, at least on simple examples. Very impressive, I must say.

There is just one snag: std.cppm is not installed. Quoting the relevant part from the above documentation:

Once libc++'s implementation is more mature we will reach out to build system vendors, with the goal that building the BMI files is done by the build system.

Currently this requires a local build of libc++ with modules enabled. Since modules are not part of the installation yet, they are used from the build directory.

In build2 we have the ability to build BMIs of external libraries on the fly (this is needed for consuming modules from installed libraries). So if std.cppm were installed somewhere where we could find it, then this would work pretty much out of the box in our case.

I can see two immediate questions about installing std.cppm: where to install it and how to make it self-contained (currently, it #includes a large number of files from the std/ subdirectory).

Regarding where to install, I would suggest just installing it next to the headers (e.g., into /usr/include/c++/v1/). I can think of two advantage with doing it this way:

1. There is already a way for a build system to find this directory (by parsing the header search paths in the -v output) and some build systems already do this (for example, in build2 we extract the compiler's header and library search paths).

2. The std.cppm file will end up in the right distribution package (e.g., -dev on Debian, -devel on Fedora, etc) without any extra effort.

The only objection I've heard to installing modules next to headers is that "it's not a header" and "it may be included by mistake". While both are true, we are already installing non-header files (e.g., inline/template implementation files) and nobody seems to be including them by mistake. We could also take an extra measure to preclude this by, for example, installing the modules into a subdirectory (e.g., /usr/include/c++/v1/modules/).

Regarding making std.cppm self-contained, I assume there is no desire to install all the 100+ files from std/ that it includes (though there is nothing technically bad about doing it, just feels unnecessary). It seems the easiest would be to just append their contents to std.cppm.in instead of #includeing them. Though I am not sure if this is easy to achieve in CMake.

Thoughts?

P.S. For completeness, let me also mention what I've done as a (hopefully) temporary measure in build2 to support import std right now. I've made a self-contained std.cppm (as described above), patched it slightly so that it works for both libc++ 17 and 18, and bundled it with build2. In other words, instead of being installed with libc++, it is being installed with build2 (again, hopefully temporarily and into build2's own directory, naturally).

@mordante @iains @ChuanqiXu9 @dwblaikie @mathstuf

[llvmbot]

@llvm/issue-subscribers-clang-modules

Author: Boris Kolpackov (boris-kolpackov)

I've been trying out the `std` module in `libc++` with `build2` (per the [Modules in libc++](https://github.com/llvm/llvm-project/blob/main/libcxx/docs/Modules.rst) documentation) and everything seems to work without any issues, at least on simple examples. Very impressive, I must say.

There is just one snag: std.cppm is not installed. Quoting the relevant part from the above documentation:

> Once libc++'s implementation is more mature we will reach out to build system vendors, with the goal that building the BMI files is done by the build system.
>
> Currently this requires a local build of libc++ with modules enabled. Since modules are not part of the installation yet, they are used from the build directory.

In build2 we have the ability to build BMIs of external libraries on the fly (this is needed for consuming modules from installed libraries). So if std.cppm were installed somewhere where we could find it, then this would work pretty much out of the box in our case.

I can see two immediate questions about installing std.cppm: where to install it and how to make it self-contained (currently, it #includes a large number of files from the std/ subdirectory).

Regarding where to install, I would suggest just installing it next to the headers (e.g., into /usr/include/c++/v1/). I can think of two advantage with doing it this way:

1. There is already a way for a build system to find this directory (by parsing the header search paths in the -v output) and some build systems already do this (for example, in build2 we extract the compiler's header and library search paths).

2. The std.cppm file will end up in the right distribution package (e.g., -dev on Debian, -devel on Fedora, etc) without any extra effort.

The only objection I've heard to installing modules next to headers is that "it's not a header" and "it may be included by mistake". While both are true, we are already installing non-header files (e.g., inline/template implementation files) and nobody seems to be including them by mistake. We could also take an extra measure to preclude this by, for example, installing the modules into a subdirectory (e.g., /usr/include/c++/v1/modules/).

Regarding making std.cppm self-contained, I assume there is no desire to install all the 100+ files from std/ that it includes (though there is nothing technically bad about doing it, just feels unnecessary). It seems the easiest would be to just append their contents to std.cppm.in instead of #includeing them. Though I am not sure if this is easy to achieve in CMake.

Thoughts?

P.S. For completeness, let me also mention what I've done as a (hopefully) temporary measure in build2 to support import std right now. I've made a self-contained std.cppm (as described above), patched it slightly so that it works for both libc++ 17 and 18, and bundled it with build2. In other words, instead of being installed with libc++, it is being installed with build2 (again, hopefully temporarily and into build2's own directory, naturally).

@mordante @iains @ChuanqiXu9 @dwblaikie @mathstuf

[ChuanqiXu9]

FYI, about the position of installing modules, there is a discussion in https://lists.isocpp.org/sg15/2023/07/1971.php

[boris-kolpackov]

BTW, to add a data point for the "perhaps the std module in libc++ is mature enough to start installing it" position: I was able to build @ChuanqiXu9 's simple_async CXX20Modules branch with libc++. All I had to do is fix a few bugs in the project and get rid of one workaround (which is not standard C++). You can see the changes here: https://github.com/boris-kolpackov/async_simple/tree/CXX20Modules-build2

[mathstuf]

Note that we don't just need the files, but also the flags to make them work as expected (namely to suppress warnings about reserved module names via -Wno-reserved-module-identifier).

[ldionne]

If we start shipping std.cppm at a given location, that will become the de-facto standard for build systems to pick up modules files from. I'm not really opposed to doing that but it seems like we'd be stepping on SG15's toes since my understanding is that we haven't come to an official agreement on that yet?

CC @Bigcheese

[mordante]

Great to hear it works with build2!

Shipping the module files is getting at the top of my todo list and I wanted to investigate which tool vendors to contact. However last week I saw @ruoso created cplusplus/modules-ecosystem-tr#29. So I prefer to participate in that effort instead of starting a similar effort. Build2 is also on that list.

Regarding the large amount of inc files. For now I intend to keep them, they are very useful for testing that libc++ exports all named declarations from a header.

[boris-kolpackov]

Note that we don't just need the files, but also the flags to make them work as expected (namely to suppress warnings about reserved module names via -Wno-reserved-module-identifier).

FWIW, this is the only extra option I had to pass to compile std.cppm. At least in build2 this whole logic is quite ad hoc: detect the importation of the std modules, notice that it's clang with libc++, find std.cppm, synthesize the dependency for the BMI, and add the extra option. And I can't see any major generalization in this area. Feels like build systems will just need to handle each compiler/standard library combination. Luckily there aren't that many of them.

I'm not really opposed to doing that but it seems like we'd be stepping on SG15's toes since my understanding is that we haven't come to an official agreement on that yet?

How can I put this politely? SG15 still hasn't come to an official agreement on the file extension for module but it seems to have agreed on the term used for compiled modules: BMI. Though there is no agreement on what it stands for: some think it's a "binary module interface", some think it's a "built module interface", there is probably another interpretation I am not aware of. So if you wait on SG15, you may end up in a situation where build systems just run out of patience and start bundling your module interfaces.

Also, FWIW, MSVC folks have decided on the location and are guaranteeing it: build2/build2#333 (comment)

Regarding the large amount of inc files. For now I intend to keep them, they are very useful for testing that libc++ exports all named declarations from a header.

Just to clarify, I didn't suggest that you get rid of them entirely, only in the installed version of std.cppm. Even that was only a suggestions -- if you decide to install them, I don't think anyone will complain.

[mathstuf]

Feels like build systems will just need to handle each compiler/standard library combination. Luckily there aren't that many of them.

There are at least a half dozen clang-based frontends with different options and no telling module flags will be one such disagreement point or not (hopefully, but some have different flag patterns they may wish to follow). It don't think it is so trivial unless one only wants to support upstream clang, Apple clang, gcc, and MSVC.

[ChuanqiXu9]

From cplusplus/modules-ecosystem-tr#29:

The first step of this process will be to collect input from a wide range of tooling providers in order to identify requirements in the ecosystem, for that we need to identify who we need to reach out to in order to get the requirements.

So it looks like that issue is not a good place to talk about the installation place for std modules.

If we want to push things forward, maybe we can send another mail to SG15 with a stronger desire to get a conclusion (instead of purely discussion), or if we want, we can make the decision ourselves. (I don't have a strong opinion here)

CC: @mordante @ldionne @boris-kolpackov

[Bigcheese]

The specific search paths used to find module source files in general hasn't been something that SG15 has discussed so far. This is largely because that very quickly hits packaging, project layout, and distribution, which are things we care about, but are large issues that would delay getting the minimum needed for modules.

I do think that discussing the limited case of where libc++ and libstdc++ should put their std module is something that's useful to discuss in SG15, and I'd be happy to setup a meeting soon to discuss it.

[boris-kolpackov]

If we want to push things forward, maybe we can send another mail to SG15 with a stronger desire to get a conclusion (instead of purely discussion), or if we want, we can make the decision ourselves.

I don't see an issue with making a decision ourselves. MSVC has already decided and if we pick something sensible I don't see a reason why libstdc++ folks won't adopt it.

I think in the end it doesn't really matter where the modules are installed provided that:

1. It doesn't cause friction for downstream package maintainers in Debian, Fedora, etc. For example, choosing something next to /usr/include/ such as /usr/modules/ will definitely cause friction.

2. It's reasonably easy for a build system to determine this location. Paths printed with -print search-dirs would be one natural place. Alternatively, something like -print-file-name=modules by analogy with -print-file-name=plugin in GCC.

The reason I personally like /usr/include/ is because it's a non-event for downstream maintainers and any serious build system already has to extract header search paths (for example, to suppress duplicates in pkg-config options, etc). But I am ok with anything else sensible.

[mathstuf]

I prefer (underneath) /usr/lib because I predict that we'll have more arch-specific files than we already have with headers necessitating some arch-ful bits in /usr/include.

[ldionne]

I encourage you (@mathstuf and @boris-kolpackov) to take part in the SG15 discussion. Let's cut a deal here. We all want to make progress on shipping modules, and I think we would all like somewhat consistent paths across implementations to keep things simple (at least I and @Bigcheese do).

Let's schedule a SG15 discussion, have it, and if there is no conclusion at the end of this SG15 discussion, then we'll do whatever we want while consulting with @jwakely to align with libstdc++. Let's at least try getting agreement within SG15 and if this doesn't seem to work, then we'll just unblock our work because I agree we need to get the ball rolling on this if we want users to start using it.

[boris-kolpackov]

I predict that we'll have more arch-specific files than we already have with headers

Do any plausible examples come to mind?

necessitating some arch-ful bits in /usr/include.

There is already a convention for installing arch-specific headers into /usr/include/<target-triplet>/, for example into /usr/include/x86_64-linux-gnu/ on my machine. We've even added an installation location name for this recently: https://github.com/build2/build2/blob/v0.16.0/NEWS#L257-L272

[jwakely]

There is already a convention for installing arch-specific headers into /usr/include/<target-triplet>/, for example into /usr/include/x86_64-linux-gnu/ on my machine.

That's specific to Debian-based distros that follow Debian's MultiArch convention. That's not universal.

[mathstuf]

Do any plausible examples come to mind?

Anything that does SIMD will have per-arch modules unless the backend is 100% isolated (which seems very unlikely to me) will expose transitive arch-specific modules. Maybe it's resolvable with preprocessor arch detection and things Just Work on Linux? Not sure how this works for macOS universal builds though.

[ChuanqiXu9]

The specific search paths used to find module source files in general hasn't been something that SG15 has discussed so far. This is largely because that very quickly hits packaging, project layout, and distribution, which are things we care about, but are large issues that would delay getting the minimum needed for modules.

I do think that discussing the limited case of where libc++ and libstdc++ should put their std module is something that's useful to discuss in SG15, and I'd be happy to setup a meeting soon to discuss it.

Can we have a meeting in 2 weeks? Since Clang/LLVM18 will be branched in the end of the next month, I really hope we can achieve this before that.

[Bigcheese]

The specific search paths used to find module source files in general hasn't been something that SG15 has discussed so far. This is largely because that very quickly hits packaging, project layout, and distribution, which are things we care about, but are large issues that would delay getting the minimum needed for modules.
I do think that discussing the limited case of where libc++ and libstdc++ should put their std module is something that's useful to discuss in SG15, and I'd be happy to setup a meeting soon to discuss it.

Can we have a meeting in 2 weeks? Since Clang/LLVM18 will be branched in the end of the next month, I really hope we can achieve this before that.

I'll send out a scheduling poll to figure out when the best time to meet is either next week or the week after. It would also be good to have some more detailed info for the other suggestions in the thread (what Boris wrote is a good level of detail for a discussion).

[fweimer-rh]

The error towards the end of https://github.com/llvm/llvm-project/blob/main/libcxx/docs/Modules.rst suggests that std.cppm itself is not portable across different compiler flags:

error: module file _deps/std-build/CMakeFiles/std.dir/std.pcm cannot be loaded due to a configuration mismatch with the current compilation [-Wmodule-file-config-mismatch]

If that's really true (and not just for compiled module representation), it's not a good idea to ship it. Before we can do that, we need something that can be used to produce a compiled module for -fexceptions, -fno-exceptions, different C++ revisions, and so on.

[Bigcheese]

The error towards the end of https://github.com/llvm/llvm-project/blob/main/libcxx/docs/Modules.rst suggests that std.cppm itself is not portable across different compiler flags:

error: module file _deps/std-build/CMakeFiles/std.dir/std.pcm cannot be loaded due to a configuration mismatch with the current compilation [-Wmodule-file-config-mismatch]

If that's really true (and not just for compiled module representation), it's not a good idea to ship it. Before we can do that, we need something that can be used to produce a compiled module for -fexceptions, -fno-exceptions, different C++ revisions, and so on.

That's for the produced BMI file, which is a binary file that is the actual compiled module interface. std.cppm is the source file from which the BMI is produced.

[mordante]

The error towards the end of https://github.com/llvm/llvm-project/blob/main/libcxx/docs/Modules.rst suggests that std.cppm itself is not portable across different compiler flags:

error: module file _deps/std-build/CMakeFiles/std.dir/std.pcm cannot be loaded due to a configuration mismatch with the current compilation [-Wmodule-file-config-mismatch]

If that's really true (and not just for compiled module representation), it's not a good idea to ship it. Before we can do that, we need something that can be used to produce a compiled module for -fexceptions, -fno-exceptions, different C++ revisions, and so on.

This is why modules need build system support, they know the user's compilation flags. Then they can create the BMI(s) that match the compiler flags. As @Bigcheese mentioned the source std.cppm is portable.

[fweimer-rh]

I see. Do we need to materialize the source file on disk? Would it be sufficient if the compiler driver had an option to build the std binary module representation according to the other compilation flags?

[boris-kolpackov]

Would it be sufficient if the compiler driver had an option to build the std binary module representation according to the other compilation flags?

You would then have to provide a mechanism for detecting when the previously compiled std module BMI is out-of-date. Also, presumably, at some point there will be the std module in libstdc++ which we would want to support. Feels like installing the std module source and letting the build system take care of the rest is the most sensible approach.

[fweimer-rh]

Would it be sufficient if the compiler driver had an option to build the std binary module representation according to the other compilation flags?

You would then have to provide a mechanism for detecting when the previously compiled std module BMI is out-of-date.

Dependency tracking options would already provide that kind of information? Just looking at the file modification time won't cover dependent files even if the source module is materialized on disk.

Also, presumably, at some point there will be the std module in libstdc++ which we would want to support. Feels like installing the std module source and letting the build system take care of the rest is the most sensible approach.

That probably needs a more complicated handshake between the projects anyway, and a single source file might not deliver all the required ingredients.

[boris-kolpackov]

@mordante Should this issues be reopened seeing that the commit in question has been reverted?

Also, I see that Clang 18 rc1 has been published which makes me wonder if the ship has sailed on having this functionality in 18. Or are you still planning/hoping to get it in before the final release?

[ChuanqiXu9]

@mordante Should this issues be reopened seeing that the commit in question has been reverted?

I don't think we should reopen this since the revered patch is in the clang part. And the job of the libc++ part, to provide option to install it, is completed.

[boris-kolpackov]

I don't think we should reopen this since the revered patch is in the clang part. And the job of the libc++ part, to provide option to install it, is completed.

This feels like splitting hairs to me since there is little use of the installed libc++ modules if nobody can find them. And to me this issue was nicely tracking the overall functionality (compared to a number of RFCs and PRs that were generated to address this) that would be needed by a build system. But Ok.

[boris-kolpackov]

I am trying to understand (and document) what ended up making it into Clang 18 specifically looking at Clang 18.1.5 in Debian (the latest release at the time of writing).

The -print-library-module-manifest-path option, it appears, did not make it:

clang++-18 -std=c++23 -stdlib=libc++ -print-library-module-manifest-path
clang++-18: error: unknown argument: '-print-library-module-manifest-path'
clang++-18: error: no input files

This seems to match #85637, which is an attempt to backport this to 18 that was abandoned.

However, the JSON file and the std/std.compat source files appear to be present:

$ cat /usr/lib/llvm-18/lib/libc++.modules.json 
{
  "version": 1,
  "revision": 1,
  "modules": [
    {
      "logical-name": "std",
      "source-path": "../share/libc++/v1/std.cppm",
      "is-std-library": true,
      "local-arguments": {
        "system-include-directories": [
          "../share/libc++/v1"
        ]
      }
    },
    {
      "logical-name": "std.compat",
      "source-path": "../share/libc++/v1/std.compat.cppm",
      "is-std-library": true,
      "local-arguments": {
        "system-include-directories": [
          "../share/libc++/v1"
        ]
      }
    }
  ]
}

ls -1 /usr/lib/llvm-18/lib/../share/libc++/v1/
std/
std.compat/
std.compat.cppm
std.cppm

Is this understanding accurate or am I missing something? If it is accurate, is it also accurate to assume that the full support for this will only be available in Clang 19?

[ChuanqiXu9]

I am trying to understand (and document) what ended up making it into Clang 18 specifically looking at Clang 18.1.5 in Debian (the latest release at the time of writing).

The -print-library-module-manifest-path option, it appears, did not make it:

clang++-18 -std=c++23 -stdlib=libc++ -print-library-module-manifest-path
clang++-18: error: unknown argument: '-print-library-module-manifest-path'
clang++-18: error: no input files

This seems to match #85637, which is an attempt to backport this to 18 that was abandoned.

However, the JSON file and the std/std.compat source files appear to be present:

$ cat /usr/lib/llvm-18/lib/libc++.modules.json 
{
  "version": 1,
  "revision": 1,
  "modules": [
    {
      "logical-name": "std",
      "source-path": "../share/libc++/v1/std.cppm",
      "is-std-library": true,
      "local-arguments": {
        "system-include-directories": [
          "../share/libc++/v1"
        ]
      }
    },
    {
      "logical-name": "std.compat",
      "source-path": "../share/libc++/v1/std.compat.cppm",
      "is-std-library": true,
      "local-arguments": {
        "system-include-directories": [
          "../share/libc++/v1"
        ]
      }
    }
  ]
}

ls -1 /usr/lib/llvm-18/lib/../share/libc++/v1/
std/
std.compat/
std.compat.cppm
std.cppm

Is this understanding accurate or am I missing something? If it is accurate, is it also accurate to assume that the full support for this will only be available in Clang 19?

Yes, I think this is accurate. BTW, maybe it is helpful to mention that, whether or not the json file gets installed depends on vendors choices.

CC @mordante and @mathstuf for double check.

[boris-kolpackov]

Another data point: I've installed Clang 19 snapshot from Debian experimental:

Debian clang version 19.0.0 (++20240421021844+e095d978ba47-1~exp1)

In this version the -print-library-module-manifest-path option is recognized but it doesn't print anything sensible:

$ clang++-19 -std=c++23 -stdlib=libc++ -print-library-module-manifest-path
<NOT PRESENT>

Even though the JSON file and the std/std.compat source files are installed, the same as for Clang 18.

[ChuanqiXu9]

Another data point: I've installed Clang 19 snapshot from Debian experimental:

Debian clang version 19.0.0 (++20240421021844+e095d978ba47-1~exp1)

In this version the -print-library-module-manifest-path option is recognized but it doesn't print anything sensible:

$ clang++-19 -std=c++23 -stdlib=libc++ -print-library-module-manifest-path
<NOT PRESENT>

Even though the JSON file and the std/std.compat source files are installed, the same as for Clang 18.

Oh, this should be a bug IIRC. @mordante. Maybe we didn't receive bug reports just because no one uses it : (

Can you compile a hello world example with -std=libc++ in this configuration?

[mathstuf]

The -print-file-name=libc++.modules.json flag works for CMake to find the file.

[boris-kolpackov]

Can you compile a hello world example with -std=libc++ in this configuration?

Yes. I can even do one better and compile it with import std; if I connect things manually.

The -print-file-name=libc++.modules.json flag works for CMake to find the file.

Hm, yes, this works for me also with Clang 19 (but not 18).

Now I am wondering what is the difference/purpose of -print-library-module-manifest-path? Can someone give an authoritative answer as to what mechanism a build system vendor should use?

[mathstuf]

I tested it with a custom-built 18.1.2, so why 18 doesn't work in your case is an interesting question…

[ChuanqiXu9]

Now I am wondering what is the difference/purpose of -print-library-module-manifest-path? Can someone give an authoritative answer as to what mechanism a build system vendor should use?

Personally I think the intention should be -print-library-module-manifest-path. But it might be better to ask @mordante

[mordante]

The manifest is indeed not installed by default in Clang 18. It will be installed by default in Clang 19. This still requires Linux distributions to add these files to their packages or provide additional packages.

The intention was indeed to use -print-library-module-manifest-path, but this indeed seems to be broken :-( It's odd, it used to work on my system. The intention is that the file could have a different name depending on compilation options.

[boris-kolpackov]

I tested it with a custom-built 18.1.2, so why 18 doesn't work in your case is an interesting question…

I also checked Clang in 18.1.1 that comes in Fedora 40:

1. -print-library-module-manifest-path is not recognized.

2. -print-file-name=libc++.modules.json doesn't print a path.

3. Neither libc++.modules.json nor std/std.compat source files appear to be present.

[mordante]

Based on Neither libc++.modules.json nor std/std.compat source files appear to be present. it seems these files are not installed on Fedora.

[boris-kolpackov]

Right, we seems to have a full spectrum of ways in which this is not working. FWIW, we are shipping the hack in build2 0.17.0 since it is currently the only way to make import std; work with Clang 18. I hope this can be fixed in time for Clang 19. But I have my doubts about making sure things are installed consistently across major distributions.

[boris-kolpackov]

@ldionne, @mordante: Is there a new milestone for making this work? Perhaps the Clang 19 release?

[ldionne]

I haven't followed the whole thread here, but our intention was for this to work in LLVM 19 out of the box. I believe we don't expect anything to work well with the 18 release. Libc++ should be fulfilling its part of the contract here, we do install the module files by default. That still requires the distributions to actually install those files as well.

If Clang misbehaves when the module files are installed properly, then that would be a Clang bug.

[llvmbot]

@llvm/issue-subscribers-clang-modules

Author: Boris Kolpackov (boris-kolpackov)

I've been trying out the `std` module in `libc++` with `build2` (per the [Modules in libc++](https://github.com/llvm/llvm-project/blob/main/libcxx/docs/Modules.rst) documentation) and everything seems to work without any issues, at least on simple examples. Very impressive, I must say.

There is just one snag: std.cppm is not installed. Quoting the relevant part from the above documentation:

> Once libc++'s implementation is more mature we will reach out to build system vendors, with the goal that building the BMI files is done by the build system.
>
> Currently this requires a local build of libc++ with modules enabled. Since modules are not part of the installation yet, they are used from the build directory.

In build2 we have the ability to build BMIs of external libraries on the fly (this is needed for consuming modules from installed libraries). So if std.cppm were installed somewhere where we could find it, then this would work pretty much out of the box in our case.

I can see two immediate questions about installing std.cppm: where to install it and how to make it self-contained (currently, it #includes a large number of files from the std/ subdirectory).

Regarding where to install, I would suggest just installing it next to the headers (e.g., into /usr/include/c++/v1/). I can think of two advantage with doing it this way:

1. There is already a way for a build system to find this directory (by parsing the header search paths in the -v output) and some build systems already do this (for example, in build2 we extract the compiler's header and library search paths).

2. The std.cppm file will end up in the right distribution package (e.g., -dev on Debian, -devel on Fedora, etc) without any extra effort.

The only objection I've heard to installing modules next to headers is that "it's not a header" and "it may be included by mistake". While both are true, we are already installing non-header files (e.g., inline/template implementation files) and nobody seems to be including them by mistake. We could also take an extra measure to preclude this by, for example, installing the modules into a subdirectory (e.g., /usr/include/c++/v1/modules/).

Regarding making std.cppm self-contained, I assume there is no desire to install all the 100+ files from std/ that it includes (though there is nothing technically bad about doing it, just feels unnecessary). It seems the easiest would be to just append their contents to std.cppm.in instead of #includeing them. Though I am not sure if this is easy to achieve in CMake.

Thoughts?

P.S. For completeness, let me also mention what I've done as a (hopefully) temporary measure in build2 to support import std right now. I've made a self-contained std.cppm (as described above), patched it slightly so that it works for both libc++ 17 and 18, and bundled it with build2. In other words, instead of being installed with libc++, it is being installed with build2 (again, hopefully temporarily and into build2's own directory, naturally).

@mordante @iains @ChuanqiXu9 @dwblaikie @mathstuf

[ChuanqiXu9]

Let me try to give a summary since it is too long and I am not sure if I understand correctly.

1. If libc++.modules.json or std/std.compat source doesn't appear, it should be the that vendors doesn't provide that.

2. If -print-file-name=libc++.modules.json works? If yes, then it should be the problem of our patch to implement -print-library-module-manifest-path and if not, there might be other problems.

Also in the case 2 is true, i suggest to file a new issue for this since it looks like a different topic than this thread.

[boris-kolpackov]

@ChuanqiXu9 I personally still don't understand whether it is -print-library-module-manifest-path or -print-file-name=libc++.modules.json that we are aiming for. @mordante suggests in an earlier comment that it's the former:

The intention was indeed to use -print-library-module-manifest-path, but this indeed seems to be broken

You seem to suggest that it's the latter (actually, you use one option in the first sentence, and the other in the second, so I am not really sure what you are suggesting, if anything).

I upgraded to the latest Clang/libc++ 19 pre-release from Debian experimental:

Debian clang version 19.0.0 (++20240529093513+2cfea14a57ad-1~exp1)

The modules are installed:

$ ls -1 /usr/lib/llvm-19/lib/../share/libc++/v1/
std
std.compat
std.compat.cppm
std.cppm

The -print-library-module-manifest-path option is recognized but does not work:

$ clang++-19 -std=c++23 -stdlib=libc++ -print-library-module-manifest-path
<NOT PRESENT>

The -print-file-name=libc++.modules.json option is recognized and appears to work:

$ clang++-19 -std=c++23 -stdlib=libc++ -print-file-name=libc++.modules.json
/usr/lib/llvm-19/bin/../lib/libc++.modules.json

So let me repeat my earlier request:

Can someone give an authoritative answer as to what mechanism a build system vendor should use?

We are almost 60 comments down in this issue, and we still can't seem to figure this out!

[ChuanqiXu9]

Can someone give an authoritative answer as to what mechanism a build system vendor should use?

We said, it should be -print-library-module-manifest-path in the end of the day. But it is broken now. And in case -print-file-name= works, it looks fine to use that first. When we fix the problem, it will be suggested to use -print-library-module-manifest-path again.

[boris-kolpackov]

@ChuanqiXu9 Ok, thanks for the clarification. Could you also explain what would be the incentive for me as a build system vendor to later switch from -print-file-name=libc++.modules.json to -print-library-module-manifest-path? It would have to be something substantial to risk destabilizing things. Maybe it makes more sense to just kill -print-library-module-manifest-path and call it a day?

[ChuanqiXu9]

@ChuanqiXu9 Ok, thanks for the clarification. Could you also explain what would be the incentive for me as a build system vendor to later switch from -print-file-name=libc++.modules.json to -print-library-module-manifest-path? It would have to be something substantial to risk destabilizing things. Maybe it makes more sense to just kill -print-library-module-manifest-path and call it a day?

To be honest, for my memory, -print-library-module-manifest-path is proposed during a SG15 meeting. So it stands for the consensus of SG15. So in the ideal world, if build system vendors follow this, they should be able to avoid some if-else to calculate the path to libc++. And if build systems don't do this, they just need to add some if-else as far as I know.