Apple Silicon toolchain support #88
Comments
|
I thought about this a bit while working on #85. I think there are actually two separate things:
I think the first can be done with the feature #85 adds but I also think it's really the second thing that you'd want when running macOS on ARM (so that To add support for arm64 darwin host platforms, we'd need to:
And I think that should be about it. I think the bigger obstacle right now is that LLVM doesn't publish binaries for arm64 macOS (even though it's definitely possible to build LLVM for arm64 macOS). This effectively means that you'd have to provide your own download URL to to Adding tests for this in CI will also be tricky since (afaik) GitHub Actions doesn't yet have ARM worker machines let alone macOS ARM worker machines but that's okay. Another question is whether we'd want to support creating universal binaries on macOS; I think it makes sense not to default to doing so (users can opt into this by setting some @jez I'm happy to try to put together these changes if you're willing to test (I don't have access to an arm64 macOS machine). |
|
I’m more than happy to test! Do you have a sense of when you’d have time to work on this? Also: I think that even being able to produce arm64 binaries would be an improvement, even if they were built by x86_64 clang. I see that #85 is a draft—is that something you’d like me to test? Or is there something else holding it back from landing? |
|
Okay! I put together a thing that uses #85 to set up a toolchain for arm64 (still x86_64 based so it's cross-compiling). It took a little doing; I'm not on macOS 11 so my sysroot didn't have the right stuff and it took me a while to realize that the # newer macOS SDKs use `.tbd` files in their sysroots; `lld` has support for this
# (on recent versions) but `ld` (that ships in the LLVM releases) does not
#
# so, we need to specify that we want to use `lld`
#
# historically, we have not done this for macOS by default because of incomplete
# support for Mach-O in `lld` but newer version seem to have good support.
"extra_linker_flags": ["-fuse-ld=lld"],I'd love to know what Apple But anyways, with the above, it does get all the way through linking on my x86_64 macOS machine. I can't actually run the generated binaries but hopefully they do actually work 🤞. I've attached the workspace I put together to this comment.
In theory running |
|
@jez re: your other questions: I should have time to try to get the #85 is still a draft because it builds on #75 which hasn't yet been merged and because it's missing some docs and polish; I wanted to solicit some feedback about some of the design choices in that PR before cleaning it up. It is more or less functional though. |
Here's the output: I'll test that workspace out now and see what happens. |
Thanks! Can you run |
|
|
is the |
|
yup, exactly |
|
I had to grab it externally because I'm on an older version of macOS that doesn't have an SDK with the right stuff to build for arm64 but for host arm64 toolchains (as in |
(I had to make a slight change to the repo, which is to make it use a script that has contents identical to this: https://github.com/jez/ragel-bison-parser-sandbox/blob/master/bazel because our company laptops prevent us from installing bazelisk) but otherwise the above is the result of running things. Interestingly enough, that's the same error I get when trying to build a normal bazel project on my macbook. For example this tiny project shows the same problems. |
|
hmm Does cc_toolchain_suite(
name = "toolchain",
toolchains = {
"k8|clang": ":cc-clang-linux",
"darwin|clang": ":cc-clang-darwin",
"k8": ":cc-clang-linux",
"darwin": ":cc-clang-darwin",
},
)for you? |
|
oh whoops, nvm; I totally missed that that's |
|
It's not clear to me why it's even analyzing |
|
|
Also in your original post you mentioned that you have an |
Yeah, overnight that seems to have stopped working. I can't explain that. |
That's odd; toolchain resolution happens exactly as we'd expect (it picks the x86 clang toolchain for macOS) but it still pulls in Can you try running
|
|
(actually the contents of |
|
oh whoops; does it get any further if you add |
|
wait ^ makes sense actually; there isn't an x86_64 darwin toolchain in does the error go away if you don't specify |
I get the same output for all combinations of is the output when |
|
😕 how about with |
|
Looks like that works? |
|
oh neat that's super weird though; I thought ^ wouldn't be necessary anymore 😕 Does just |
|
Neither of those works |
|
You might also be curious to see the cquery output with Click to expand |
|
thanks, I was just about to ask 😛 |
|
that's super weird; maybe something leftover from the regardless – I'm glad it works! even though it really does seem like you shouldn't have to pass in Just to be sure: does actually running the binary that's produced work? |
|
I've been working on compiling the Sorbet project using the provided ARM64 toolchains. It seems that the playground workspace changes work as intended, but there is one thing missing for me to be able to fully verify the build completes and works. The only change I had to make to the playground code was setting Also, despite being on the most recent macos, I still have to use The last remaining error is when linking we get the error
Would it be possible to rebase that playground version with the current LLVM 13 support that is already in main? Do you believe that is likely the cause of frameworks not working or could it be something else that I am missing? I did try to use the current version in main to try to upgrade to LLVM 13, but doing so fails with this error, so maybe we need something else too. Note: another thing worth noting is that, even when using the playground, we must sign the resulting binary with |
|
@vinistock which playground workspace are you referring to? Do you have a fork/branch that you could share? |
I think they're referring to the workspace attached in this comment. |
Hmm. Are there maybe other toolchains registered by the Sorbet workspace? Can you verify that the
This is odd; lld has definitely had support for
Not sure what's going on here; Maybe this has finally become an issue? It'd be helpful to compare against Sorbet on regular x86_64 macOS with the same toolchain (if that builds successfully).
The playground is based on #85;
I'm not really sure what's going on but I don't think it's an LLVM version issue.
The version of this repo currently in
My understanding is that @jez do you remember if you needed to explicitly sign the binaries you got out of Bazel? Regardless, I think this is very much in scope for @vinistock Thanks for the notes! I now have access to an Apple Silicon machine; I'm hoping to get back to working on this issue early next week. If possible, can you post your modified workspace/the commands you are running? |
|
@rrbutani thank you so much for the detailed and quick response. I really appreciate your assistance. Debugging the toolchainsI'm not exactly sure how I should be reading this output. Please, let me know if you need more information. But running It seems to reject the toolchain, but then selects it afterwards. Not sure if there's an issue in toolchain resolution here. Comparing with x86With the same style of configuration of the playground, if I use Playground exampleThe playground example I meant is In this playground repo, if the link option for framework is present I cannot compile with the error If I remove the Notice that in this playground repo, I can also compile the x86 version successfully even with the framework flag (exactly as in Sorbet). The command I'm using to compile is # Successfully compiles even with the -framework Foundation linkopt
bazel build //:test --config=x86
# Fails with `unable to find framework Foundation`
bazel build //:test --config=arm64 |
|
Since we determined that we'll need LLVM 13 for signing binaries, I have begun looking into what we'll need to upgrade Sorbet to LLVM 13, using the latest version of The upgrade is relatively smooth, but I bumped into another issue that I'm having trouble figuring out. Please, let me know if this is not within the scope of What the upgrade consisted ofBasically, there were two steps
The bugAfter the upgrade, I was able to determine that this invocation to find_cpp_toolchain started returning the wrong toolchain. In version 12, the toolchain object we get has all the LLVM paths in it (includes, With version 13, the latest In addition to returning the wrong toolchain, I also noticed that all paths are now relative, whereas when using version 12 they are all absolute. Attempts to fix itIn the Bazel issue linked in the README describing the migration for I tried doing the migration, making sure that the rules depended on the right toolchains and that we were now using the new I believe this might be related to the Any ideas on why the LLVM toolchain is not being returned by |
This is a little out of scope for this project but that's fine.
It sounds like toolchain resolution is giving you back the toolchain installed on your machine instead of the one from Did you also remember to add a call to In case you haven't already come across it, these docs do a good job explaining toolchains, toolchain resolution, and how rulesets should use toolchains. For C/C++ toolchains the actual toolchain lookup ( Making the changes described in ^ (i.e. using Building a target that uses that rule with
I'm fairly confident this is unrelated; There used to be a bug in the |
|
@rrbutani once again, thank you for the quick and detailed response. I indeed had forgotten to invoke I'm now hitting one last error before successfully compiling the custom Ruby build. One of Ruby's arguments during compilation is .../llvm_toolchain_13_0_0/bin/cc_wrapper.sh: line 54: executable_path/../lib/libruby.2.7.dylib: No such file or directoryThe reason this happens is because the .../cc_wrapper ... -install_name @execution_path/../lib/libruby.2.7.dylib -o libruby.2.7.dylibNotice that the file not existing only fails because Do you have any context as to why that is necessary or if we can workaround it somehow? |
No worries! Glad to hear it was a simple fix.
This is a good catch! That snippet was added to support parameter files. The macOS cc wrapper script inspects the full command line in order to remap libraries that are being linked against to their fully resolved paths, taking into account the Anyways, for that reason we need to actually read what's in the parameter file. The PR in this repo you linked to was essentially copied from upstream (this commit); in general the logic in the macOS wrapper mostly comes from upstream. The issue here, of course, is that the What's peculiar to me is that upstream Bazel seems to fail on this in the exact same way (here's a minimal test case). Perhaps it's simply not common for users to want to generate dylibs with I think extending the logic in the macOS wrapper to skip processing args starting with I have a few concerns though:
@vinistock If possible, it'd be super helpful if you could point me to where in your workspace that flag is getting added. |
|
Actually, I'm going to open a new issue for this. |
|
I worked through some more investigations on this and thought I'd bring up another thing I'm seeing. I'm trying to change a dependency of a package by matching on a The code I have on a Sorbet branch is equivalent to this: # BUILD file
platform(
name = "darwin_arm",
constraint_values = [
"@platforms//cpu:arm64",
"@platforms//os:osx",
]
)
platform(
name = "darwin_x86",
constraint_values = [
"@platforms//cpu:x86_64",
"@platforms//os:osx",
],
)
config_setting(
name = "darwin",
constraint_values = [
"@platforms//cpu:x86_64",
"@platforms//os:osx",
]
)
config_setting(
name = "darwin_arm64",
constraint_values = [
"@platforms//cpu:arm64",
"@platforms//os:osx",
]
)
# Library BUILD file
cc_library(
...
deps = select({
"//:darwin": ["x86_specific_dependency"],
"//:darwin_arm64": ["portable_version_dependency"],
"//conditions:default": ["portable_version_dependency"]
})
)However, we always match This is the Sorbet branch I'm currently playing with if that helps. |
|
@vinistock make sure that you are using a bazel arm64 release and not bazel amd64 running on rosetta. I think there is also a flag that help debug toolchain resolution that can be passed to bazel. |
|
@sluongng you are right! I was indeed running an x86 version of bazel. Upgrading to 5.0.0 with an ARM64 executable seems to fix not only the issue with conditionally compiling the dependency, but also the I will spend some time validating it, but it looks like that was the issue all along. |
|
I can confirm that using the ARM64 version of Bazel fixes the I am, however, having the same issue I was having before of the custom Ruby build selecting the wrong toolchains when using While debugging, I noticed that the arm64 toolchain is rejected by using the I was also able to confirm that changing the toolchain(
name = "clang-darwin-arm64",
exec_compatible_with = [
"@platforms//cpu:x86_64", # If this value is x86_64, the toolchain is rejected. If we use arm64, the toolchain is selected
"@platforms//os:osx",
],
target_compatible_with = [
"@platforms//cpu:arm64",
"@platforms//os:osx",
],
toolchain = ":clang-darwin-arm64-toolchain",
toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
)I'm not sure why, when using |
|
Hi! Just wondering if any progress has been made on that since February? |
|
LLVM 15.0.0 was released, which contains 'clang+llvm-15.0.0-arm64-apple-darwin21.0.tar.xz'. |
|
I checked it out and it works really nice! (https://github.com/omerbenamram/bazel-toolchain/tree/llvm-15.0.0) |
We've been using the bazel-toolchain library so we can use our custom LLVM toolchain. This was working perfectly on CI because we compile on Linux there. On our local M1 machines this was falling back on our xcode compiler instead of the custom toolchain because bazel-toolchain doesn't have support for the darwin-arm64 configuration. There is an existing GitHub issue about this (bazel-contrib#88) with a lot of activity but nothing landed. There also seemed to be some outstanding PRs, but they are complete rewrites; and it's not clear when they would land. My guess is that we may be able to drop our patches at some point, but we shouldn't wait for it. The main changes here are as follows: * We add support for a darwin-arm64 toolchain to bazel-toolchain. * This requires some additional configuration to handle the various ridiculous goop that macOS builds need; bazel already has toolchain support for this, so most of it is copied from the bazelbuild repo. It would be great to upstream this, but that might be a significant effort. * It appears that bazel-toolchain handled everything using the `unix_cc_toolchain_config`, which only had limited supported for Objective C. Since we need to fully be able to compile objective C applications, we need to include `osx_cc_toolchain_config` as well. * The `osx_cc_toolchain_config` uses a clang wrapper called `wrapped_clang` and `wrapped_clang_pp`. This is written as a C++ source file for some reason, so it needs to be compiled as part of the toolchain build. This wrapper forwards to xcode clang/clang++ by default, but we want to forward to our own toolchain instead. So we have to modify the wrapper slightly to look at some environment variables to determine where the real toolchain is. * The clang compiler includes some directories by default, specifically ones that are included with the compiler. This works if you run the toolchain manually, but bazel doesn't like this because it wants to make sure that it knows about all included headers. Normally you should be able to use the `cxx_builtin_include_directories` attribute to notify bazel about these directories, but it doesn't seem to work. It seems that the problem may be that the included paths are absolute instead of relative, so they don't appear to be the same path to bazel. Using clang's `-no-canonical-prefixes` flag is meant to fix this, but it didn't seem to work for me. As an alternate workaround, I made sure to include all the built-in directories again via `-isystem` so that the paths match exactly.
15.0.2 does have a couple of x86 linux builds. i tested the RHEL on an ubuntu machine and it worked fine, so maybe we could just use that? more artifacts are listed in the discourse thread for the release. off version 0.7.2 of this repo, a patch like this seems to work (props to @omerbenamram for basically putting this diff together): diff --git a/toolchain/cc_toolchain_config.bzl b/toolchain/cc_toolchain_config.bzl
index 8785a8e..1ac65a5 100644
--- a/toolchain/cc_toolchain_config.bzl
+++ b/toolchain/cc_toolchain_config.bzl
@@ -68,6 +68,15 @@ def cc_toolchain_config(
"clang",
"darwin_x86_64",
"darwin_x86_64",
+ ),
+ "darwin-aarch64": (
+ "clang-aarch64-darwin",
+ "aarch64-apple-macosx",
+ "darwin",
+ "macosx",
+ "clang",
+ "darwin_aarch64",
+ "darwin_aarch64",
),
"linux-aarch64": (
"clang-aarch64-linux",
diff --git a/toolchain/internal/common.bzl b/toolchain/internal/common.bzl
index 7493c64..53b3b53 100644
--- a/toolchain/internal/common.bzl
+++ b/toolchain/internal/common.bzl
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-SUPPORTED_TARGETS = [("linux", "x86_64"), ("linux", "aarch64"), ("darwin", "x86_64")]
+SUPPORTED_TARGETS = [("linux", "x86_64"), ("linux", "aarch64"), ("darwin", "x86_64"), ("darwin", "aarch64")]
host_tool_features = struct(
SUPPORTS_ARG_FILE = "supports_arg_file",
@@ -68,7 +68,10 @@ def arch(rctx):
])
if exec_result.return_code:
fail("Failed to detect machine architecture: \n%s\n%s" % (exec_result.stdout, exec_result.stderr))
- return exec_result.stdout.strip()
+ arch = exec_result.stdout.strip()
+ if arch == "arm64":
+ return "aarch64"
+ return arch
def os_arch_pair(os, arch):
return "{}-{}".format(os, arch)
diff --git a/toolchain/internal/llvm_distributions.bzl b/toolchain/internal/llvm_distributions.bzl
index 074ed84..43473ae 100644
--- a/toolchain/internal/llvm_distributions.bzl
+++ b/toolchain/internal/llvm_distributions.bzl
@@ -207,6 +207,12 @@ _llvm_distributions = {
"clang+llvm-14.0.0-x86_64-apple-darwin.tar.xz": "cf5af0f32d78dcf4413ef6966abbfd5b1445fe80bba57f2ff8a08f77e672b9b3",
"clang+llvm-14.0.0-x86_64-linux-gnu-ubuntu-18.04.tar.xz": "61582215dafafb7b576ea30cc136be92c877ba1f1c31ddbbd372d6d65622fef5",
"clang+llvm-14.0.0-x86_64-linux-sles12.4.tar.xz": "78f70cc94c3b6f562455b15cebb63e75571d50c3d488d53d9aa4cd9dded30627",
+
+ # 15.0.2
+ "clang+llvm-15.0.2-arm64-apple-darwin21.0.tar.xz": "8c33f807bca56568b7060d0474daf63c8c10ec521d8188ac76362354d313ec58",
+ "clang+llvm-15.0.2-x86_64-apple-darwin.tar.xz": "a37ec6204f555605fa11e9c0e139a251402590ead6e227fc72da193e03883882",
+ "clang+llvm-15.0.2-aarch64-linux-gnu.tar.xz": "527ed550784681f95ec7a1be8fbf5a24bd03d7da9bf31afb6523996f45670be3",
+ "clang+llvm-15.0.2-x86_64-unknown-linux-gnu-rhel86.tar.xz": "f48f479e91ee7297ed8306c9d4495015691237cd91cc5330d3e1ee057b0548bd",
}
# Note: Unlike the user-specified llvm_mirror attribute, the URL prefixes in
@@ -229,6 +235,7 @@ _llvm_distributions_base_url = {
"13.0.0": "https://github.com/llvm/llvm-project/releases/download/llvmorg-",
"13.0.1": "https://github.com/llvm/llvm-project/releases/download/llvmorg-",
"14.0.0": "https://github.com/llvm/llvm-project/releases/download/llvmorg-",
+ "15.0.2": "https://github.com/llvm/llvm-project/releases/download/llvmorg-",
}
def _get_auth(ctx, urls):
diff --git a/toolchain/tools/llvm_release_name.py b/toolchain/tools/llvm_release_name.py
index 39505cc..3485d61 100755
--- a/toolchain/tools/llvm_release_name.py
+++ b/toolchain/tools/llvm_release_name.py
@@ -30,7 +30,12 @@ def _patch_llvm_version(llvm_version):
def _darwin(llvm_version, arch):
major_llvm_version = _major_llvm_version(llvm_version)
- suffix = "darwin-apple" if major_llvm_version == 9 else "apple-darwin"
+ if major_llvm_version == 9:
+ suffix = "darwin-apple"
+ elif arch == "arm64":
+ suffix = "apple-darwin21.0"
+ else:
+ suffix = "apple-darwin"
return "clang+llvm-{llvm_version}-{arch}-{suffix}.tar.xz".format(
llvm_version=llvm_version, arch=arch, suffix=suffix)
@@ -86,6 +91,8 @@ def _linux(llvm_version, distname, version, arch):
# If you find this mapping wrong, please send a Pull Request on Github.
if arch in ["aarch64", "armv7a", "mips", "mipsel"]:
os_name = "linux-gnu"
+ elif major_llvm_version == 15:
+ os_name = "unknown-linux-gnu-rhel86"
elif distname == "freebsd":
os_name = "unknown-freebsd-%s" % version
elif distname == "suse": |
|
It looks like 15.0.3 was recently released, but unfortunately still does not include any x86_64 linux binaries. For now, I'm used the patch provided by @iamricard and @omerbenamram. It seems to work well, with the addition of an extra patch to fix the strip_prefix for the I have an example standalone project using this toolchain here. It's focused on building cross-platform py3_image targets, so the use of this toolchain is really orthogonal to the purpose of the demo, but it may be useful to someone. Full patch, in case that repo gets removed at some point: |
|
I rebased @omerbenamram's change on top of the recent commits from this repo supporting 15.06 if anybody wants to use it: https://github.com/meastham/bazel-toolchain/tree/m1_support Seems to be working fine |
|
@meastham can you open a PR? |
|
Sure: #174 |
I just got an M1 MacBook Pro today, and am looking into how to use this project to generate arm64 binaries (for the record: everything works fine using this project to generate x86_64 binaries, which then run under Rosetta).
In the Apple Developer Documentation, they make it out to be as simple as passing a
-targetflag to the C compiler, though I'm sure it'll be more work to do the same thing in Bazel.https://developer.apple.com/documentation/apple-silicon/building-a-universal-macos-binary
Have anyone put thought or time into how this project might be extended to support generating arm64 binaries on M1 Macs? I'm probably going to be spending some time getting this working, and I'd love any tips, ideas, or places to start.
The text was updated successfully, but these errors were encountered: