Set up Mac builds with clang >= 17 even though Xcode only has at most clang 16

[swolchok]

🚀 The feature, motivation and pitch

This would enable a couple disparate improvements:

1. Our binary releases should include the latest compiler features and optimizations. The concrete motivating example is that the compiler used for Mac wheels apparently doesn't pass COMPILER_SUPPORTS_BF16_TARGET (i.e., clang version greater than 15), which causes a slower bfloat16 gemv kernel to be used.
2. We should have test coverage for CPU bfloat16 support on Mac (We don't have test coverage for aarch64 (ARM64) bfloat16 feature (__ARM_FEATURE_BF16) #142703) -- clang 16 purports to be able to build it, but is buggy and we actually need 17+.

Alternatives

do nothing until Apple gets around to releasing an Xcode with clang 17 or later and we get around to updating to it.

Additional context

Xcode clang version history: https://gist.github.com/yamaya/2924292 . Latest at time of writing is Xcode 16.2 with Apple clang version 16.0.0 (clang-1600.0.26.6)

cc @seemethere @malfet @osalpekar @atalman @pytorch/pytorch-dev-infra

[Skylion007]

As a nit: since clang is so new on Mac builds, might be a good candidate to experiment with enabling LTO on. 👍

[malfet]

Thank you for creating the issue. Few thoughts:

Our binary releases should include the latest compiler features and optimizations.

PyTorch binary wheels should be installable on wide range of OSes customer might have. If latest compiler does not enforce newer binary compatibility, than sure, I believe we should do that. But also, one should asses how hard will it be to maintain this custom compiler build and build in expertise triaging issues that it might trigger compared to the stock compilers.

• Though for Linux, we could/should have at least one CI config that tests latest clang available
• For MacOS, imo it's better to stick to what Apple officially releases

Detecting compiler features thru ifdefs is not the Cmake way, one should test if something like that is supported by the current compiler by trying to compile it locally. I.e. I can compile the BFDOT16 code locally without a problem using stock Xcode compiler

[huydhn]

We don't plan to take any action for this right now and will wait for Apple to release a new clang/xcode version on MacOS

[huydhn]

As @malfet points out, it looks like there might be a route to unblock this using the beta MacOS 15 runner from GitHub https://github.com/actions/runner-images/blob/main/images/macos/macos-15-arm64-Readme.md. If I read it correctly, it offers clang 18 via homebrew