Add layering_check support for macOS #22259

keith · 2024-05-06T20:11:27Z

There were 2 things with the previous implementation that needed to be improved here:

Apple Clang has a bug where it doesn't pass module compiler flags to the underlying -cc1 invocation, so we have to manually pass them directly to that invocation with -Xclang
The previous search script was too aggressive and slow for macOS. The macOS SDK has tons of files that aren't headers, and tons of symlinks pointing to other files within the SDK. This adds a fork in the script to run a version that works with Apple SDKs. The time difference on my machine is 41s->6s. 6s is still pretty long so if desired we can put this behavior behind an env var for users to opt in with.

I've added a hermetic version of this to the apple_support toolchain, but similar to the Linux setup here the modulemap file includes absolute paths.

fmeum

You need to run this target to update the lockfile:

bazel/src/test/tools/bzlmod/BUILD

Line 12 in 3bc70b2

name = "update_default_lock_file",

Looks good! The faster find wouldn't work on Linux, so the amount of "fork"ing looks just right.

keith · 2024-05-14T20:39:14Z

@pzembrod could you take a look? 🙏🏻

pzembrod · 2024-05-15T10:29:50Z

@keith I left a few comments on your push, because apparently GitHub's UI succeeded in confusing me. ;-)

comius · 2024-05-15T13:34:21Z

@keith I left a few comments on your push, because apparently GitHub's UI succeeded in confusing me. ;-)

I can't find the comments, but I'm ok if @keith can.

brentleyjones · 2024-05-15T13:49:41Z

@comius here: e59f370

tools/cpp/unix_cc_toolchain_config.bzl

pzembrod · 2024-05-15T13:52:09Z

tools/cpp/unix_cc_configure.bzl

@@ -534,7 +534,7 @@ def configure_unix_toolchain(repository_ctx, cpu_value, overriden_tools):
        ["/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk"],
    )

-    generate_modulemap = is_clang and not darwin


This was disabled before on MacOS because it was too slow?

tools/cpp/generate_system_module_map.sh

pzembrod · 2024-05-15T13:52:18Z

src/test/shell/bazel/bazel_layering_check_test.sh

@@ -144,11 +144,6 @@ EOF
 }

 function test_bazel_layering_check() {


Just checking: the related test/feature test_bazel_layering_check_header_only() still doesn't work on MacOS?

i didn't want to change it in this PR too since it's unrelated, but I think we can enable it afterwards #22370

pzembrod · 2024-05-15T13:52:56Z

I moved my comments over to the PR

There were 2 things with the previous implementation that needed to be improved here: 1. Apple Clang has a bug where it doesn't pass module compiler flags to the underlying -cc1 invocation, so we have to manually pass them directly to that invocation with -Xclang 2. The previous search script was too aggressive and slow for macOS. The macOS SDK has tons of files that aren't headers, and tons of symlinks pointing to other files within the SDK. This adds a fork in the script to run a version that works with Apple SDKs. The time difference on my machine is 41s->6s. 6s is still pretty long so if desired we can put this behavior behind an env var for users to opt in with. I've added a hermetic version of this to the apple_support toolchain, but similar to the Linux setup here the modulemap file includes absolute paths.

pzembrod · 2024-05-17T08:28:59Z

src/test/shell/bazel/bazel_layering_check_test.sh

@@ -144,11 +144,6 @@ EOF
 }

 function test_bazel_layering_check() {


meteorcloudy · 2024-05-21T15:31:04Z

This seems to break Bazel postsubmit: https://buildkite.com/bazel/bazel-bazel/builds/28021#018f9a8b-8711-4f02-af8a-c29a1e7c6255

//src/test/shell/bazel:bazel_bootstrap_distfile_tar_test

# Configuration: 7c706e29d44fb244031cd9f1b5563076a031b073c9a89bd361b0680404c6068b
# Execution platform: //:default_host_platform
�[1mexternal/grpc~/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc:42:10: �[0m�[0;1;31merror: �[0m�[1mmodule grpc~//:grpc_resolver_dns_ares does not depend on a module exporting 'ares.h'�[0m
#include <ares.h>
�[0;1;32m         ^
�[0m1 error generated.
�[32m[1,873 / 4,221]�[0m 36 actions running

keith · 2024-05-21T16:16:23Z

I think we can either patch gRPC to disable the layering check for that target, or we can try and update it (not sure how hard that is for bazel?) wdyt?

keith · 2024-05-21T17:12:49Z

submitted #22468 because the upgrade is non-trivial and requires protocolbuffers/protobuf#16884

keith · 2024-05-21T17:13:30Z

@bazel-io flag

*** Reason for rollback *** Broke Bazel postsubmit: https://buildkite.com/bazel/bazel-bazel/builds/28021#018f9a8b-8711-4f02-af8a-c29a1e7c6255 #22259 (comment) *** Original change description *** Add layering_check support for macOS There were 2 things with the previous implementation that needed to be improved here: 1. Apple Clang has a bug where it doesn't pass module compiler flags to the underlying -cc1 invocation, so we have to manually pass them directly to that invocation with -Xclang 2. The previous search script was too aggressive and slow for macOS. The macOS SDK has tons of files that aren't headers, and tons of symlin... *** RELNOTES: PiperOrigin-RevId: 635852793 Change-Id: I03b0831706a555bbcf141742aefa3bf48ed0caf3

iancha1992 · 2024-05-21T18:19:27Z

@bazel-io fork 7.2.0

There were 2 things with the previous implementation that needed to be improved here: 1. Apple Clang has a bug where it doesn't pass module compiler flags to the underlying -cc1 invocation, so we have to manually pass them directly to that invocation with -Xclang 2. The previous search script was too aggressive and slow for macOS. The macOS SDK has tons of files that aren't headers, and tons of symlinks pointing to other files within the SDK. This adds a fork in the script to run a version that works with Apple SDKs. The time difference on my machine is 41s->6s. 6s is still pretty long so if desired we can put this behavior behind an env var for users to opt in with. I've added a hermetic version of this to the apple_support toolchain, but similar to the Linux setup here the modulemap file includes absolute paths. Closes bazelbuild#22259. PiperOrigin-RevId: 635735410 Change-Id: Icb3f30dd7319f147b0db54b3b42d4dc9032dc1a0

There were 2 things with the previous implementation that needed to be improved here: 1. Apple Clang has a bug where it doesn't pass module compiler flags to the underlying -cc1 invocation, so we have to manually pass them directly to that invocation with -Xclang 2. The previous search script was too aggressive and slow for macOS. The macOS SDK has tons of files that aren't headers, and tons of symlinks pointing to other files within the SDK. This adds a fork in the script to run a version that works with Apple SDKs. The time difference on my machine is 41s->6s. 6s is still pretty long so if desired we can put this behavior behind an env var for users to opt in with. I've added a hermetic version of this to the apple_support toolchain, but similar to the Linux setup here the modulemap file includes absolute paths. Closes bazelbuild#22259. This reverts commit 1f1b4fd.

keith · 2024-05-21T18:33:35Z

got reverted, new version here #22475

There were 2 things with the previous implementation that needed to be improved here: 1. Apple Clang has a bug where it doesn't pass module compiler flags to the underlying -cc1 invocation, so we have to manually pass them directly to that invocation with -Xclang 2. The previous search script was too aggressive and slow for macOS. The macOS SDK has tons of files that aren't headers, and tons of symlinks pointing to other files within the SDK. This adds a fork in the script to run a version that works with Apple SDKs. The time difference on my machine is 41s->6s. 6s is still pretty long so if desired we can put this behavior behind an env var for users to opt in with. I've added a hermetic version of this to the apple_support toolchain, but similar to the Linux setup here the modulemap file includes absolute paths. Closes bazelbuild#22259. This reverts commit 1f1b4fd.

There were 2 things with the previous implementation that needed to be improved here: 1. Apple Clang has a bug where it doesn't pass module compiler flags to the underlying -cc1 invocation, so we have to manually pass them directly to that invocation with -Xclang 2. The previous search script was too aggressive and slow for macOS. The macOS SDK has tons of files that aren't headers, and tons of symlinks pointing to other files within the SDK. This adds a fork in the script to run a version that works with Apple SDKs. The time difference on my machine is 41s->6s. 6s is still pretty long so if desired we can put this behavior behind an env var for users to opt in with. I've added a hermetic version of this to the apple_support toolchain, but similar to the Linux setup here the modulemap file includes absolute paths. Closes #22259. This reverts commit 1f1b4fd. Partial commit for third_party/*, see #22475. Change-Id: I801121e36de1504c17adfa4736c49c88d470fec0 Signed-off-by: Hee Cha <heec@google.com>

There were 2 things with the previous implementation that needed to be improved here: 1. Apple Clang has a bug where it doesn't pass module compiler flags to the underlying -cc1 invocation, so we have to manually pass them directly to that invocation with -Xclang 2. The previous search script was too aggressive and slow for macOS. The macOS SDK has tons of files that aren't headers, and tons of symlinks pointing to other files within the SDK. This adds a fork in the script to run a version that works with Apple SDKs. The time difference on my machine is 41s->6s. 6s is still pretty long so if desired we can put this behavior behind an env var for users to opt in with. I've added a hermetic version of this to the apple_support toolchain, but similar to the Linux setup here the modulemap file includes absolute paths. Closes bazelbuild#22259. This reverts commit 1f1b4fd.

There were 2 things with the previous implementation that needed to be improved here: 1. Apple Clang has a bug where it doesn't pass module compiler flags to the underlying -cc1 invocation, so we have to manually pass them directly to that invocation with -Xclang 2. The previous search script was too aggressive and slow for macOS. The macOS SDK has tons of files that aren't headers, and tons of symlinks pointing to other files within the SDK. This adds a fork in the script to run a version that works with Apple SDKs. The time difference on my machine is 41s->6s. 6s is still pretty long so if desired we can put this behavior behind an env var for users to opt in with. I've added a hermetic version of this to the apple_support toolchain, but similar to the Linux setup here the modulemap file includes absolute paths. Closes #22259. This reverts commit 1f1b4fd. Closes #22475. PiperOrigin-RevId: 637969838 Change-Id: I7d4940a820e3741836239493222ba8d06c4d70e4

There were 2 things with the previous implementation that needed to be improved here: 1. Apple Clang has a bug where it doesn't pass module compiler flags to the underlying -cc1 invocation, so we have to manually pass them directly to that invocation with -Xclang 2. The previous search script was too aggressive and slow for macOS. The macOS SDK has tons of files that aren't headers, and tons of symlinks pointing to other files within the SDK. This adds a fork in the script to run a version that works with Apple SDKs. The time difference on my machine is 41s->6s. 6s is still pretty long so if desired we can put this behavior behind an env var for users to opt in with. I've added a hermetic version of this to the apple_support toolchain, but similar to the Linux setup here the modulemap file includes absolute paths. Closes bazelbuild#22259. This reverts commit 1f1b4fd. Closes bazelbuild#22475. PiperOrigin-RevId: 637969838 Change-Id: I7d4940a820e3741836239493222ba8d06c4d70e4

There were 2 things with the previous implementation that needed to be improved here: 1. Apple Clang has a bug where it doesn't pass module compiler flags to the underlying -cc1 invocation, so we have to manually pass them directly to that invocation with -Xclang 2. The previous search script was too aggressive and slow for macOS. The macOS SDK has tons of files that aren't headers, and tons of symlinks pointing to other files within the SDK. This adds a fork in the script to run a version that works with Apple SDKs. The time difference on my machine is 41s->6s. 6s is still pretty long so if desired we can put this behavior behind an env var for users to opt in with. I've added a hermetic version of this to the apple_support toolchain, but similar to the Linux setup here the modulemap file includes absolute paths. Closes bazelbuild#22259. This reverts commit 1f1b4fd. Partial commit for third_party/*, see bazelbuild#22475. Change-Id: I801121e36de1504c17adfa4736c49c88d470fec0 Signed-off-by: Hee Cha <heec@google.com>

There were 2 things with the previous implementation that needed to be improved here: 1. Apple Clang has a bug where it doesn't pass module compiler flags to the underlying -cc1 invocation, so we have to manually pass them directly to that invocation with -Xclang 2. The previous search script was too aggressive and slow for macOS. The macOS SDK has tons of files that aren't headers, and tons of symlinks pointing to other files within the SDK. This adds a fork in the script to run a version that works with Apple SDKs. The time difference on my machine is 41s->6s. 6s is still pretty long so if desired we can put this behavior behind an env var for users to opt in with. I've added a hermetic version of this to the apple_support toolchain, but similar to the Linux setup here the modulemap file includes absolute paths. Closes #22259. This reverts commit 1f1b4fd. Partial commit for third_party/*, see #22475. Change-Id: I801121e36de1504c17adfa4736c49c88d470fec0 Signed-off-by: Hee Cha <heec@google.com> Commit de0a37c Signed-off-by: Hee Cha <heec@google.com> Co-authored-by: Keith Smiley <keithbsmiley@gmail.com>

There were 2 things with the previous implementation that needed to be improved here: 1. Apple Clang has a bug where it doesn't pass module compiler flags to the underlying -cc1 invocation, so we have to manually pass them directly to that invocation with -Xclang 2. The previous search script was too aggressive and slow for macOS. The macOS SDK has tons of files that aren't headers, and tons of symlinks pointing to other files within the SDK. This adds a fork in the script to run a version that works with Apple SDKs. The time difference on my machine is 41s->6s. 6s is still pretty long so if desired we can put this behavior behind an env var for users to opt in with. I've added a hermetic version of this to the apple_support toolchain, but similar to the Linux setup here the modulemap file includes absolute paths. Closes bazelbuild#22259. This reverts commit 1f1b4fd. Closes bazelbuild#22475. PiperOrigin-RevId: 637969838 Change-Id: I7d4940a820e3741836239493222ba8d06c4d70e4

There were 2 things with the previous implementation that needed to be improved here: 1. Apple Clang has a bug where it doesn't pass module compiler flags to the underlying -cc1 invocation, so we have to manually pass them directly to that invocation with -Xclang 2. The previous search script was too aggressive and slow for macOS. The macOS SDK has tons of files that aren't headers, and tons of symlinks pointing to other files within the SDK. This adds a fork in the script to run a version that works with Apple SDKs. The time difference on my machine is 41s->6s. 6s is still pretty long so if desired we can put this behavior behind an env var for users to opt in with. I've added a hermetic version of this to the apple_support toolchain, but similar to the Linux setup here the modulemap file includes absolute paths. Closes #22259. This reverts commit 1f1b4fd. Closes #22475. PiperOrigin-RevId: 637969838 Change-Id: I7d4940a820e3741836239493222ba8d06c4d70e4 Commit 6abdd2a Co-authored-by: Keith Smiley <keithbsmiley@gmail.com>

keith requested a review from fmeum May 6, 2024 20:11

github-actions bot added team-Rules-CPP Issues for C++ rules awaiting-review PR is awaiting review from an assigned reviewer labels May 6, 2024

fmeum approved these changes May 6, 2024

View reviewed changes

keith force-pushed the ks/add-layering_check-support-for-macos branch 6 times, most recently from 67c225a to d0cdc1d Compare May 6, 2024 22:07

keith force-pushed the ks/add-layering_check-support-for-macos branch from d0cdc1d to e59f370 Compare May 14, 2024 20:03

comius added awaiting-user-response Awaiting a response from the author and removed awaiting-review PR is awaiting review from an assigned reviewer labels May 15, 2024

pzembrod reviewed May 15, 2024

View reviewed changes

keith force-pushed the ks/add-layering_check-support-for-macos branch from e59f370 to 0286a53 Compare May 15, 2024 19:24

keith requested a review from pzembrod May 15, 2024 19:24

pzembrod approved these changes May 17, 2024

View reviewed changes

src/test/shell/bazel/bazel_layering_check_test.sh

@@ -144,11 +144,6 @@ EOF

}

function test_bazel_layering_check() {

Copy link

Contributor

pzembrod May 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm

pzembrod added awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally and removed awaiting-user-response Awaiting a response from the author labels May 17, 2024

copybara-service bot closed this in 0a65893 May 21, 2024

github-actions bot removed the awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally label May 21, 2024

keith deleted the ks/add-layering_check-support-for-macos branch May 21, 2024 16:27

keith mentioned this pull request May 21, 2024

Disable broken layering_check for grpc #22468

Closed

bazel-io added the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label May 21, 2024

bazel-io removed the potential release blocker Flagged by community members using "@bazel-io flag". Should be added to a release blocker milestone label May 21, 2024

bazel-io mentioned this pull request May 21, 2024

[7.2.0] Add layering_check support for macOS #22472

Closed

bazel-io mentioned this pull request May 21, 2024

[7.2.0] Add layering_check support for macOS #22474

Closed

keith mentioned this pull request May 21, 2024

Add layering_check support for macOS #22475

Closed

bazel-io mentioned this pull request May 28, 2024

[7.2.0] Add layering_check support for macOS #22572

Merged

bazel-io mentioned this pull request May 28, 2024

[7.2.0] Add layering_check support for macOS #22575

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add layering_check support for macOS #22259

Add layering_check support for macOS #22259

keith commented May 6, 2024

fmeum left a comment

keith commented May 14, 2024

pzembrod commented May 15, 2024

comius commented May 15, 2024

brentleyjones commented May 15, 2024

pzembrod May 15, 2024

keith May 15, 2024

pzembrod May 15, 2024

keith May 15, 2024

pzembrod May 17, 2024

pzembrod commented May 15, 2024

pzembrod May 17, 2024

meteorcloudy commented May 21, 2024

keith commented May 21, 2024

keith commented May 21, 2024

keith commented May 21, 2024

iancha1992 commented May 21, 2024

keith commented May 21, 2024

		@@ -144,11 +144,6 @@ EOF
		}

		function test_bazel_layering_check() {

Add layering_check support for macOS #22259

Add layering_check support for macOS #22259

Conversation

keith commented May 6, 2024

fmeum left a comment

Choose a reason for hiding this comment

keith commented May 14, 2024

pzembrod commented May 15, 2024

comius commented May 15, 2024

brentleyjones commented May 15, 2024

pzembrod May 15, 2024

Choose a reason for hiding this comment

keith May 15, 2024

Choose a reason for hiding this comment

pzembrod May 15, 2024

Choose a reason for hiding this comment

keith May 15, 2024

Choose a reason for hiding this comment

pzembrod May 17, 2024

Choose a reason for hiding this comment

pzembrod commented May 15, 2024

pzembrod May 17, 2024

Choose a reason for hiding this comment

meteorcloudy commented May 21, 2024

keith commented May 21, 2024

keith commented May 21, 2024

keith commented May 21, 2024

iancha1992 commented May 21, 2024

keith commented May 21, 2024