Add a filegroup containing _all_ sources to the libc build rules #169155
Conversation
These rules already expose a filegroup containing the _dependencies_, but that misses the source files directly in the top level library. Without this filegroup, there isn't a way to access the source files used by libcxx when building it, etc.
chandlerc
requested review from
aaronmondal,
keith and
rupprecht
as code owners
|
@llvm/pr-subscribers-libc Author: Chandler Carruth (chandlerc) ChangesThese rules already expose a filegroup containing the dependencies, but that misses the source files directly in the top level library. Without this filegroup, there isn't a way to access the source files used by libcxx when building it, etc. That said, if there is a better way to do this or a different design, let me know. Not super familiar with the Bazel logic for llvm-libc. Full diff: https://github.com/llvm/llvm-project/pull/169155.diff 1 Files Affected:
diff --git a/utils/bazel/llvm-project-overlay/libc/libc_build_rules.bzl b/utils/bazel/llvm-project-overlay/libc/libc_build_rules.bzl
index 0f2965369c296..c871334dd95b3 100644
--- a/utils/bazel/llvm-project-overlay/libc/libc_build_rules.bzl
+++ b/utils/bazel/llvm-project-overlay/libc/libc_build_rules.bzl
@@ -247,6 +247,12 @@ def libc_header_library(name, hdrs, deps = [], **kwargs):
**kwargs
)
+ _libc_srcs_filegroup(
+ name = name + "_hdrs",
+ libs = [":" + name],
+ enforce_headers_only = True,
+ )
+
def libc_generated_header(name, hdr, yaml_template, other_srcs = []):
"""Generates a libc header file from YAML template.
|
|
Could you clarify what/how you're trying to build? E.g. we have an llvm-libc BUILD target (cc_library) here: https://github.com/llvm/llvm-project/blob/main/utils/bazel/llvm-project-overlay/libc/BUILD.bazel#L1817-L1831 that is used in downstream Bazel build of libcxx (it's simply listed as a dependency of proper rules for libcxx). |
I'm trying to assemble a package with Bazel that has all the sources needed to build libc++, not trying to build libc++ itself. The goal is to allow Carbon to support building runtimes on-demand in all cases, so we're installing all the source files and building in build steps that we'll run automatically as needed. From some discussions with other llvm-libc folks, it seemed like using the sources directly was an intended and supported use case (and that matches having the file groups for the dependencies), so I was sending this PR to somewhat complete the picture. But can also keep this out of tree if needed. I don't quite have the last patches for libc++ polished, but happy to point folks to them once they are. |
Noted. Do you also assemble a package with the sources needed to build llvm-libc? If yes, do you use smth. like @jtstogel planned to look into this, and explore an idea of adding a separate step of auto-generating BUILD.bazel |
I've not looked at shipping libc itself yet, just libc++ and libunwind. Long term, definitely interested, but libc++ was specifically blocking work and so was prioritizing that. The current dependency isn't particularly noticeable for me yet. |
This builds on the previous work to flesh out more on-demand runtimes building. It adds building of the `libc++.a` archive runtime. A number of changes are required for this to work: - The runtimes build infrastructure needs to support building sources from multiple parts of LLVM rather than a single part. We do this by lifting the root of the runtimes source paths up a level to a common runtimes tree, and installing the runtimes sources below this directory. - Both libc++ and libc++abi runtimes sources need to be installed, and we even need to install some interesting parts of llvm-libc that are used in the build of libc++. - We need to generate the site configuration header file for libc++ from the CMake template. This includes both setting up a set of platform-independent defines and introducing some basic Bazel support for processing the CMake template itself. Doing all of this also exposed some missing features and limitations of the runtimes building infrastructure that are addressed here. One note is that all of this just adds libc++ to the explicit `build-runtimes` command for testing. It doesn't yet trigger automatically building these prior to linking, or configuring any of the other subcommands to automatically use these runtimes. All of that will come in follow-up PRs. Also, this makes the `clang_runtimes_test` ... _very_ slow in our default build configuration. Compiling libc++, even with many threads on a large Linux server requires up to 50 seconds. I'm open to any suggestions on how to handle this, including disabling the test in non-optimized builds. I have some ideas to speed this up, but fundamentally building libc++ is... not cheap. I did look at some of the existing Bazel tools to process the CMake template, but they all seemed significantly more complex than what we need and didn't have broad adoption. Given that, it seemed slightly better to just roll our own given the simple format. The second new LLVM patch is currently under review upstream and so hopefully temporary: llvm/llvm-project#169155
|
FYI, a PR patching this in and using it is here: carbon-language/carbon-lang#6424 |
This builds on the previous work to flesh out more on-demand runtimes building. It adds building of the `libc++.a` archive runtime. A number of changes are required for this to work: - The runtimes build infrastructure needs to support building sources from multiple parts of LLVM rather than a single part. We do this by lifting the root of the runtimes source paths up a level to a common runtimes tree, and installing the runtimes sources below this directory. - Both libc++ and libc++abi runtimes sources need to be installed, and we even need to install some interesting parts of llvm-libc that are used in the build of libc++. - We need to generate the site configuration header file for libc++ from the CMake template. This includes both setting up a set of platform-independent defines and introducing some basic Bazel support for processing the CMake template itself. Doing all of this also exposed some missing features and limitations of the runtimes building infrastructure that are addressed here. One note is that all of this just adds libc++ to the explicit `build-runtimes` command for testing. It doesn't yet trigger automatically building these prior to linking, or configuring any of the other subcommands to automatically use these runtimes. All of that will come in follow-up PRs. Also, this makes the `clang_runtimes_test` ... _very_ slow in our default build configuration. Compiling libc++, even with many threads on a large Linux server requires up to 50 seconds. I'm open to any suggestions on how to handle this, including disabling the test in non-optimized builds. I have some ideas to speed this up, but fundamentally building libc++ is... not cheap. I did look at some of the existing Bazel tools to process the CMake template, but they all seemed significantly more complex than what we need and didn't have broad adoption. Given that, it seemed slightly better to just roll our own given the simple format. The second new LLVM patch is currently under review upstream and so hopefully temporary: llvm/llvm-project#169155
This builds on the previous work to flesh out more on-demand runtimes building. It adds building of the `libc++.a` archive runtime. A number of changes are required for this to work: - The runtimes build infrastructure needs to support building sources from multiple parts of LLVM rather than a single part. We do this by lifting the root of the runtimes source paths up a level to a common runtimes tree, and installing the runtimes sources below this directory. - Both libc++ and libc++abi runtimes sources need to be installed, and we even need to install some interesting parts of llvm-libc that are used in the build of libc++. - We need to generate the site configuration header file for libc++ from the CMake template. This includes both setting up a set of platform-independent defines and introducing some basic Bazel support for processing the CMake template itself. Doing all of this also exposed some missing features and limitations of the runtimes building infrastructure that are addressed here. One note is that all of this just adds libc++ to the explicit `build-runtimes` command for testing. It doesn't yet trigger automatically building these prior to linking, or configuring any of the other subcommands to automatically use these runtimes. All of that will come in follow-up PRs. Also, this makes the `clang_runtimes_test` ... _very_ slow in our default build configuration. Compiling libc++, even with many threads on a large Linux server requires up to 50 seconds. I'm open to any suggestions on how to handle this, including disabling the test in non-optimized builds. I have some ideas to speed this up, but fundamentally building libc++ is... not cheap. I did look at some of the existing Bazel tools to process the CMake template, but they all seemed significantly more complex than what we need and didn't have broad adoption. Given that, it seemed slightly better to just roll our own given the simple format. The second new LLVM patch is currently under review upstream and so hopefully temporary: llvm/llvm-project#169155
This builds on the previous work to flesh out more on-demand runtimes building. It adds building of the `libc++.a` archive runtime. A number of changes are required for this to work: - The runtimes build infrastructure needs to support building sources from multiple parts of LLVM rather than a single part. We do this by lifting the root of the runtimes source paths up a level to a common runtimes tree, and installing the runtimes sources below this directory. - Both libc++ and libc++abi runtimes sources need to be installed, and we even need to install some interesting parts of llvm-libc that are used in the build of libc++. - We need to generate the site configuration header file for libc++ from the CMake template. This includes both setting up a set of platform-independent defines and introducing some basic Bazel support for processing the CMake template itself. Doing all of this also exposed some missing features and limitations of the runtimes building infrastructure that are addressed here. One note is that all of this just adds libc++ to the explicit `build-runtimes` command for testing. It doesn't yet trigger automatically building these prior to linking, or configuring any of the other subcommands to automatically use these runtimes. All of that will come in follow-up PRs. Also, this makes the `clang_runtimes_test` ... _very_ slow in our default build configuration. Compiling libc++, even with many threads on a large Linux server requires up to 50 seconds. I'm open to any suggestions on how to handle this, including disabling the test in non-optimized builds. I have some ideas to speed this up, but fundamentally building libc++ is... not cheap. I did look at some of the existing Bazel tools to process the CMake template, but they all seemed significantly more complex than what we need and didn't have broad adoption. Given that, it seemed slightly better to just roll our own given the simple format. The second new LLVM patch is currently under review upstream and so hopefully temporary: llvm/llvm-project#169155
3 participants
These rules already expose a filegroup containing the dependencies, but that misses the source files directly in the top level library.
Without this filegroup, there isn't a way to access the source files used by libcxx when building it, etc.
That said, if there is a better way to do this or a different design, let me know. Not super familiar with the Bazel logic for llvm-libc.