Implementation of Metal backend

[philipturner]

This is a progress tracker for implementation of the hipSYCL backend. I figured it's most appropriate to extract it into a separate thread, and not flood #460.

Preliminary Research:

• Completing GROMACS optimizations unrelated to hipSYCL
• Exploring MoltenCL, decided not to pursue that path
• Exploring SPIR-V -> AIR transpiler, requires investing too much time

Milestones:

• Bare-bones hipSYCL backend
• Feature-complete hipSYCL backend

Planned Features:

• USM pointers (all types)
• atomic64 emulation
• Optimized subgroup/workgroup reductions

Optional Features:

• float64 emulation
• Simdgroup matrix and shuffle-and-fill instructions
• Port to iOS

[FCLC]

I was thinking about this topic whilst trying to get HipSYCL going via both brew and SPACK.

Specifically on the topic of a metal transpiler, is it worth looking at the Khronos MoltenVK project? https://github.com/KhronosGroup/MoltenVK

I'd briefly brought up Sylkan as more of a place holder than anything in #839, with the main issues being around Spir-V (amongst others).

As for FP64, during a discussion on Mastodon with Karol Herbst of RedHat on the topic of a Mesa SYCL implementation using RUSTICL and the problems he was also facing around FP64, I'd made the suggestion of using/adapting the MIT licensed code already available from Intel in their OpenCL compute runtime. (I'm having trouble finding the discussion, will edit it in when I come across it)

[philipturner]

MIT licensed code already available from Intel in their OpenCL compute runtime. (I'm having trouble finding the discussion, will edit it in when I come across it)

Based on my (incomplete) understanding, Intel uses a sub-optimal algorithm to emulate FP64, possibly the -soft-float already in LLVM. I'm looking at LLVM's code and Berkeley SoftFloat, then trying to create a higher-performance alternative with better accuracy. Optimization this requires extreme fine-tuning at the assembly level, such as subdividing a 53-bit mantissa into odd sub-components, then multiplying in a way that allows transferring carry bits between intermediate sums. The end result is something ~2-4x faster than LLVM's code.

The issue with MoltenVK and Sylkan is they transpile SPIR-V to MSL. This has some restrictions mentioned in #460, such as requiring structured control flow. We can't have those restrictions if we want to implement hipSYCL. We need to lower directly to AIR, which shouldn't be that difficult; I just have to familiarize myself with LLVM bitcode. Some also recently reverse-engineered the Metallib format.

That being said, related work by other people can contribute good ideas. Thanks for your suggestions!

[FCLC]

As for FP64, during a discussion on Mastodon with Karol Herbst of RedHat on the topic of a Mesa SYCL implementation using RUSTICL and the problems he was also facing around FP64, I'd made the suggestion of using/adapting the MIT licensed code already available from Intel in their OpenCL compute runtime. (I'm having trouble finding the discussion, will edit it in when I come across it)

I managed to track down the thread, it's here: https://chaos.social/@FelixCLC@mastodon.social/109387984784067583

The relevant pieces of code are here:
https://github.com/intel/compute-runtime/blob/6cbb3cfb0563406574a48c50dd202fefb7e6e4cd/shared/source/ail/ail_configuration_extra.cpp#L16-L23
https://github.com/intel/intel-graphics-compiler/blob/master/IGC/Compiler/CISACodeGen/Emu64OpsPass.cpp
With a better read in of what is going on here:
https://github.com/intel/intel-graphics-compiler/blob/master/documentation/visa/2_datatypes.md#fundamental-data-types

The issue with MoltenVK and Sylkan is they transpile SPIR-V to MSL.

Sounds good! Is there an active repo-fork? From your account I see some projects such as https://github.com/philipturner/air-v and https://github.com/philipturner/metal-float64 which seem to be relevant. I'm assuming the plan is to eventually port/bring some of that work into hipSYCL as dependencies for a metal backend?

[philipturner]

The relevant pieces of code are here:

Thanks. That's more reference material to help me build MetalFloat64.

Sounds good! Is there an active repo-fork? From your account I see some projects such as https://github.com/philipturner/air-v and https://github.com/philipturner/metal-float64 which seem to be relevant. I'm assuming the plan is to eventually port/bring some of that work into hipSYCL as dependencies for a metal backend?

Correct. I have to start with the fundamentals, and it's progressing at a snail's pace. I will have significant free time over winter break and hope to finish metal-float64 then.

[philipturner]

Just redesigned the approach to USM, and it has much more capabilities (working POC at metal-usm). It can access all of device RAM, often with zero overhead. USM shared allocations are allowed, and in fact the only true allocation type. System malloc pointers are not supported though.

@illuhad looking at libfloor and Metal.jl, we don't actually need an LLVM fork. Julia used a custom LLVM toolchain, but we could just as easily access LLVM functions outside the LLVM code base (like you do). That includes copying the LLVM 5.0 bitcode writer, which I don't think we'll need anyway. We can write .air files and let xcrun metallib package them into a Metallib file. That approach also lets us optimize with metal-opt.

If I could make a direct llvm-air instead of mediating through SPIR-V, that would save a massive amount of time. The runtime code for interfacing with the Metal API (e.g. MTL::Device, MTL::CommandQueue) would already be baked into hipSYCL's source tree. We need an M1 device to test the the code regardless. Implementing the LLVM IR -> AIR translator out-of-tree won't make it any easier to test or maintain.

[woachk]

Hello,

which I don't think we'll need anyway

You will need it. Please don't skip that part.

If I could make a direct llvm-air instead of mediating through SPIR-V, that would save a massive amount of time

And will result in something significantly more brittle.

[from an earlier message]

such as requiring structured control flow.

That's the kind of thing that is better handled via a conversion pass IMO anyway. Structured control flow is for example also necessary for Vulkan's SPIR-V dialect.

You might also want to see the Mesa codebase in this case. NIR is built around structured control flow. See what rusticl does for example, or clvk too (which has a lot of its problems due to not having VK_KHR_buffer_device_address, which isn't an issue here).

We need to lower directly to AIR

No you don't (due to the above) :-)

[from the first one]

USM pointers (all types)

USM/SVM is the important part for SW compat.

A possible strategy: You might want to evaluate possibilities of doing a hardcoded mmap() to the gpuAddress. This might work but is not guaranteed and will break at some point...

My 2 cents: You might want to entertain starting with rusticl on MoltenVK. It's as good of a starting point as any...

[philipturner]

@woachk thanks for your advice. I just started experimenting with LLVM and AIR. In a C++ program importing LLVM 15 headers, I read a simple .air file through parseModule, then re-serialized it into another file. I passed the new file into xcrun metallib and it compiled just fine, then ran on the GPU without bugs. I was thinking that we modify this workflow for hipSYCL. Instead of getting our original llvm::Module from an .air file, we get the llvm::Module from the hipSYCL LLVM IR that SSCP pass 1 generates. However, if structurizing control flow is accomplishable through a simple LLVM function, perhaps we could code-gen Metal Shader source ("llvm-to-msl").

A possible strategy: You might want to evaluate possibilities of doing a hardcoded mmap() to the gpuAddress. This might work but is not guaranteed and will break at some point...

That's exactly* how I redesigned my approach to USM. It opens up all 32 GB of device RAM without needing to statically allocate everything beforehand. Here's the source code. It seems that every time you deallocate a MTLBuffer, the subsequent allocation recycles its address (given you don't change allocation size). This might not be true if another process allocates something at the same time. However, it should take a finite number of tries to correct the CPU offset to match the GPU VA.

* The CPU base address is not actually the same as the GPU base address. It must be offset by a factor of 16 for some reason. The addresses don't have to be equal, just translatable through an integer addition.

And will result in something significantly more brittle.

Having to constantly stay alert for bugs is a downside. It might always work only on the newest version of macOS. However, I have practically zero free time these days. Cutting corners might be the only way this happens in the near future.

[woachk]

xcrun metallib and it compiled just fine

xcrun metallib is a linker, it does not check AIR correctness.

However, if structurizing control flow is accomplishable through a simple LLVM function

Structured control flow is mandatory for a number of targets, from Vulkan SPIR-V to JavaScript and WebAssembly. As such, that part of the LLVM code is well-tested (because of the latter).

Having to constantly stay alert for bugs is a downside. It might always work only on the newest version of macOS. However, I have practically zero free time these days. Cutting corners might be the only way this happens in the near future.

That's fine, it's just something that you should be aware of. :)

[philipturner]

Structured control flow is mandatory for a number of targets, from Vulkan SPIR-V to JavaScript and WebAssembly. As such, that part of the LLVM code is well-tested (because of the latter).

The direct LLVM->MSL route also runs trivially run iOS (MSL can compile without Metal command-line tools), and it's easier to debug. I should have just went with this original idea all along!

[woachk]

Structured control flow is mandatory for a number of targets, from Vulkan SPIR-V to JavaScript and WebAssembly. As such, that part of the LLVM code is well-tested (because of the latter).

The direct LLVM->MSL route also runs trivially run iOS (MSL can compile without Metal command-line tools), and it's easier to debug. I should have just went with this original idea all along!

If you want to reuse existing code, rusticl on Vulkan (which does LLVM IR -> OpenCL SPIR-V [unstructured] -> NIR [structurization pass included on the way in] -> Vulkan SPIR-V [part of Zink] -> Zink -> Vulkan driver) might be worth looking at.

You can also see when clspv runs the structurize pass for a starting point: https://github.com/google/clspv/blob/83055c38842e19d4dd45dc1812e88570a10f8598/lib/Compiler.cpp#L626.

[woachk]

Structured control flow is mandatory for a number of targets, from Vulkan SPIR-V to JavaScript and WebAssembly. As such, that part of the LLVM code is well-tested (because of the latter).

The direct LLVM->MSL route also runs trivially run iOS (MSL can compile without Metal command-line tools), and it's easier to debug. I should have just went with this original idea all along!

The quickest path would be to do LLVM -> structured SPIR-V to leverage the existing MoltenVK SPIR-V Cross investment: https://github.com/KhronosGroup/SPIRV-Cross. This can radically reduce the required development effort for this one.

[philipturner]

Alright, sounds like a plan. I will have to modify SPIRV-Cross to add more compute instructions, and slightly change how it generates MSL (for FP64 and USM).

[woachk]

Alright, sounds like a plan. I will have to modify SPIRV-Cross to add more compute instructions, and slightly change how it generates MSL (for FP64 and USM).

Preferably for USM what you want is for the VAs to match exactly on each side. Might be brittle on some OS versions but is probably the best case if you can get it...

[philipturner]

Preferably for USM what you want is for the VAs to match exactly on each side. Might be brittle on some OS versions but is probably the best case if you can get it...

That's not going to be possible, because the GPU's address space (0x0000001500000000) is always unallocatable in mmap. Rather, the addresses differ by a constant offset. Most often, pointers originate from the hardware buffer table (function arguments) or are indirectly captured from the immediate C++ scope. We can translate those from CPU -> GPU while encoding a command.

The nonzero overhead is from nested data structures. In some cases, you'll fetch a pointer once and read from it multiple times, with different offsets. The MSL compiler might detect this and optimize away repeated translations. Worst case, 4 cycles penalty still doesn't affect maximum bandwidth.