Enabling Cross-platform Support

[Titus-von-Koeller]

This is a more informal discussion thread that allows us to exchange ideas around the current effort to make bitsandbytes cross-platform. We want this to be community-driven and make it so that the community can gradually add functionality for their given platform, while not hindering new developments + refactorings in the library.

For doing this we would like to take good design decisions that the community is on board with and come up with a plan to enable cross-platform CI/CD for quick iteration.

[rickardp]

Some questions

• Which cross platform build tool should be used for the native code? There are multiple PRs open to use CMake but @Titus-von-Koeller also mentions there are some issues with choosing CMake. What alternatives exist, and can we make a decision which tool to use? Need input from @TimDettmers
• Should the CUDA libraries be built into the binary wheels? They make the package quite large. An alternative is to move them into external packages and use e.g. bitsandbytes[cuda12] to install Bitsandbytes with the CUDA 12 kernels. On the other hand, that would constitute a breaking change downstreams, and maybe the size (~60 MB per CUDA version IIRC, not sure for other targets like AMD) is acceptable?
• How do we continuously test the CUDA code? The MPS (Apple) code can be tested on hosted runners, but IIRC, the CUDA code requires large runners which is a GH enterprise feature. This is probably also true for AMD and other platforms which are not supported by the standard GitHub runners (I am not sure about intel). Also, there is no hosted ARM64 runner, but ARM64 features like Neon can be tested in emulation (although performance will suffer).
• Should we use a build system for Python, such poetry?

[Titus-von-Koeller]

Thanks @rickardp for the valuable inputs. Some of these things I was already going to address in a series of issues around enabling cross-platform support that I'm going to publish and other things you mentioned are new valuable input.

The issues aim at rallying everyone interested in order to agree on a set of requirements, solution approach and coordinate the community contributions. Hopefully I'll finish and publish them today, I've been working on gathering info for some days now, so your inputs came at just the right time.

To get back to you about your question from #918 (sorry, it took me a while to react, as there were the holidays, I was sick for 2 weeks and then needed to focus on getting back up to speed with the FSDP + QLoRA collab):

Yes, you can depend on us pushing through with the cross-platform collab: At least to the point that, among other priorities, we will support all PRs in order to put the right infra in place to make this possible and make sure the implementations are correct. We would like an approach were the individual algorithms of the backend can be gradually or even only partially implemented by whomever is willing to put in the work. However, it has been decided that the maintenance team will take on more of a supportive role in enabling community contributions and that the backend implementations themselves will have to be fully community contributed.

Seems Intel is currently putting significant resources to allow for an Intel backend in BNB. We're grateful for this work, happy to enable it and also make sure this to happens in a way that works well for other hardware backends and community contributors, especially since there has been such big interest on these topics before and they have unfortunately gone unaddressed so far.

We aim to have bitsandbytes be an active community repo. I'm full-time maintainer now and @younesbelkada part-time. We're supported by HF as part of their effort to support the eco-system. Therefore, you can be quite sure that your work will not be at risk of hitting a dead end. However, Tim will be largely unavailable in the next 2-4 months, as he's currently interviewing for academic positions.

For the other topics you mentioned, let me get back to those in the respective cross-platform issues that I'm gonna open.

Thanks again for your willingness to contribute and we're very happy to still have you onboard! Looking forward to work together :)

P.S. For building I think CMake is likely gonna be chosen (we already have Tim's buy-in), based on the fact that it seems very suitable for the use-case (cross-platform + easy GH actions integration) and the existing work of you, @niclimcy, @wkpark and others.

For reference, Tim's most recent reply on choosing the right device abstraction (which would profit from community input).

[matthewdouglas]

Should the CUDA libraries be built into the binary wheels? They make the package quite large. An alternative is to move them into external packages and use e.g. bitsandbytes[cuda12] to install Bitsandbytes with the CUDA 12 kernels. On the other hand, that would constitute a breaking change downstreams, and maybe the size (~60 MB per CUDA version IIRC, not sure for other targets like AMD) is acceptable?

@rickardp This is a good point as the wheels could quickly grow out of control. I'm not personally super well-versed in the Python packaging ecosystem (but so far it seems like a rough landscape). Here's my thinking:

• We're at least going to need to start building separate wheels for each platform: manylinux/x86_64, manylinux/aarch64, win/amd64, macosx/arm64, and maybe macosx/x86_64
• Are we sure we need prebuilt binaries in the same wheels for all the different CUDA versions or would it be acceptable to provide binaries for just a subset?
• For backwards compatibility, maybe it makes sense to continue to include the CUDA binaries in the default linux wheels, but other backends in extras?

[rickardp]

We're at least going to need to start building separate wheels for each platform: manylinux/x86_64, manylinux/aarch64, win/amd64, macosx/arm64, and maybe macosx/x86_64
Yes, absolutely, I think this is a non discussion. Packaging all platforms in a single wheel is not doable as this project uses native modules. Besides the fact that this is what binary wheels were designed for.

For backwards compatibility, maybe it makes sense to continue to include the CUDA binaries in the default linux wheels, but other backends in extras?
That is an interesting idea, but which CUDA version in this case?

[Titus-von-Koeller]

I agree that we should go with binary wheels. The issue that people are installing BNB and it's not working for Mac and Windows is also because we publish it as a cross-platform package, which makes no sense.

@akx Also said we could request an increase in the file size limit. I put that on my todo. In that case we could keep the CUDA binaries together. The binary wheel platform distinction and file naming convention doesn't really account for the need of different CUDA versions.. At the same time, it's kind of a waste to send so much traffic around for delivering stuff that people don't actually need. They need just a single CUDA version right? Maybe we should also have an install selector like PyTorch.

[akx]

PyTorch has the luxury of having a team of engineers working on delivering PyTorch in various builds and versions, and the infrastructure to run the separate download.pytorch.org index for the huge wheels they need to ship. I think bitsandbytes will need to be a bit leaner.

I think a neat approach to explore here would be:

• bitsandbytes itself is a cross-platform pure-Python package with no native code
• bitsandbytes-cuda110, bitsandbytes-cuda11x, bitsandbytes-cuda12x (etc.; I'm assuming those version families would work) are platform-specific binary packages (delivered as wheels) that only contain the .dll and .so file, and minimal Python packaging (likely just a bitsandbytes_cuda110/__init__.py).
• A future bitsandbytes-mps (Metal Performance Shaders) is a darwin-aarch64 only wheel
• ... and repeated for other future architectures.

Package extras could be set up in bitsandbytes to allow easy installation of the dependent library, e.g. pip install bitsandbytes[cuda110] would do the work required.

Then, upon initialization, bitsandbytes would discover which implementation the user desires. A fancy way would be entrypoints, but to be honest, I think just looking through the known backend module names in order of likelihood while allowing e.g. BNB_BACKEND to override it would be enough.

If there is no available backend, BNB would tell the user how to get one for their hardware. If the backend fails to initialize, it could have instructions on how to fix things.

[rickardp]

Thank you for the elaborate answer. There seems to be a quite large community around the porting effort and it is good to hear that people are contributing on their day job too.

We would like an approach were the individual algorithms of the backend can be gradually or even only partially implemented by whomever is willing to put in the work

I 100% agree with this and this is what I wanted to champion in my initial PRs. IMHO, this effort need to be supported by a good test suite, so PRs can quickly
Be evaluated and to lower the barrier of entry for contributors.

I also think one of the first implementations should be a correct, portable CPU implementation to serve as a baseline/reference for contributors. This would also make a partial implementation “usable” in theory (where a gradually increasing percentage of the code would be accelerated).

[matthewdouglas]

@rickardp

• How do we continuously test the CUDA code? The MPS (Apple) code can be tested on hosted runners, but IIRC, the CUDA code requires large runners which is a GH enterprise feature. This is probably also true for AMD and other platforms which are not supported by the standard GitHub runners (I am not sure about intel). Also, there is no hosted ARM64 runner, but ARM64 features like Neon can be tested in emulation (although performance will suffer).

This is relevant: https://github.blog/changelog/2024-01-30-github-actions-introducing-the-new-m1-macos-runner-available-to-open-source/

[Titus-von-Koeller]

That's great. HF agreed to support us for CUDA + Mac runners. If you two would set things up to use these runners, we could go ahead with this. For CUDA + Linux runners it's a bit more complicated, because we would have to self-host them and spin them up on demand in the cloud.

[rickardp]

That's really good, but it is not needed for building. In PR #949, macOS is already cross compiling for both M1 and x64, and can run MPS tests. Though it would be really useful with M1 to unit test on the M1 platform as it cannot be emulated.

If HF can support with CUDA agents I think that's a viable plan then.

[Titus-von-Koeller]

The questions is just how to best setup the CUDA runners, because we're going to self-host them.. The costs can also easily add up there. The questions is also if we need Windows + GPU runners. That would add quite a lot of cost, especially if we run on every commit.

[Titus-von-Koeller]

Hey all, I decided that all these efforts, what's still needed to implement + discuss is so wide ranging that I would like to have some more structure. I created a sort of meta RFC, linking to all the sub-topics, but it's still a bit of a work in progress.

I would like your support for the following:

@wkpark Could you please open a new issue with the title [RFC] Cross-Platform Refactor: Windows support?

There you would summarize all the steps already there and others that are needed to get Windows support fully working and merged. We can then edit the post (you and I can both edit the issue description) to update about the status quo of what's still needed.

I just want to make sure that we discuss things in a central place rather than all over the place and that we have one post/issue per topic where we summarize the status quo and todos as well as coordination.

Same @rickardp for the Mac topic. That would be super helpful 🤗

[wkpark]

Windows support is much simpler than macOS. windows already supports cuda and we can use it but macos does not. (I think we don't have to extract a separate issue but I can summarize the current state of Windows support)

• windows dll detection (merged)

• fix to compile under windows PR minimal patch to fix Windows compilation issues #876 (ready to merge merged)

  • std::thread support for win32
  • AVX related fix

• build script (cmake + workflow fix) PR Cmake + workflows #908 (cmake build using pip+conda+mamba without docker images) (merged😍)

See also

• Cmake + workflows #908 (comment)
• Cmake + workflows #908 (review)

[Titus-von-Koeller]

Hey @wkpark,

cool, what you wrote is perfect. Thanks a lot! Exactly what I was thinking.

In which order would you merge the Cmake workflows?

I'm very happy to hear that you think that Windows support is much more straight-forward. I'll make sure to help you as much as I can, so that we can have that implemented + merged soon. One important topic that I feel is still a blocker is running CI for all platforms that we support. So having a CI with GPU runners for Linux + Windows is still the biggest blocker for this IMO. If we don't have that, accepting community contributions will become much harder going forward. Another issue are the flaky tests. I'll create issues for discussion of these cross-platform topics today (I've been working on this for a few days now, there was a lot of activity in this repo to catch up on and take into account).

Regarding having a separate issue: This cross-platform topic is very hard for us maintainers to stay on top of with the current discussion being spread among so many places without a central hub and a single place, for each sub-topic of the refactor (e.g. CI and testing) where we summarize the current status, decisions + open actions (and where help is wanted/needed).

Therefore, I would like to have one issue for each sub-topic of the cross-platform refactor and link them via the "central-hub" crossplatform issue that I linked above.

Since you have been the champion of driving forward Windows support, I would like you to create the issue, as you can then also edit the post and be the lead on this topic. As an admin, I can edit the post as well. That way we can always keep the top-most post up-to-date with the current summary + status together. From there we also link to all the efforts + issues that are related and extract relevant bits to keep track of, so they don't get lost.

This would make it much, much easier for us to enable you best. It's been quite a lot of work to catch up with all the efforts and discussions and going forward, we need a bit of structure in order to make things easily graspable and actionable for both maintainers + the community.

Would you please create the issue [RFC] Cross-Platform Refactor: Windows support or instead of "Windows support" part, feel free to choose another title with Windows in it. What you wrote here in your last comment would then be posted as the issue text. I will then link it to the overview "central hub" issue.

[Titus-von-Koeller]

@wkpark @rickardp I would like to add you to the official BNB collaborator Slack. Would you be interested? I would like to keep as much as we can public, but sometimes it helps to discuss things via messaging, especially for coordination, and then post summaries for the community where applicable.

If you don't want to post here, you can send them to me at my firstname at huggingface dot co.

[rickardp]

Sorry, I've been completely bogged down by other stuff the last couple of days. I sent you an e-mail regarding the Slack channel earlier. I agree that Slack is a better form of communication as it if faster.

I will rebase my PR now. There's less and less left as @wkpark already integrated large part of it. Awesome work!

I'll create two RFCs that I think are relevant, one for MacOS and one for targeting CPU only

[rickardp]

I created #1020 and #1021

[Titus-von-Koeller]

@akx Since you're input has been so helpful and I see that you care about the future of the library: I'm wondering if you have any opinion on the PR for the abstraction that would allow different device backends. This is quite an important strategic design decision, that will also affect the other community contributions, e.g. AMD, Mac, Windows.

Would be very interested to hear your opinion, but if it's not sth you want to look into right now, no worries.

[akx]

Sure – I took a quick look and the concept looks sound, but needs a bit of polishing. See my comment over there. 😄

[Titus-von-Koeller]

Thanks for your review + comment. I totally agree on the Abc.

[niclimcy]

Testing each backend could be as simple as writing a pytest fixture for each backend perhaps?

[akx]

@niclimcy Even better, a single parametrizing fixture can yield multiple backends. See eg. https://github.com/python-babel/babel/blob/master/tests/conftest.py#L26

[Titus-von-Koeller]

Cross-platform support is a thing we're very actively working right now and I just wanted to quickly ping everyone that I saw had an interest in contributing on these topics when going through all old issues and PRs on these topics.

Please see the issue that acts as a central hub of the topic for reference of the current status and the open topics that would need community input + contributions to make this happen.

So sorry that a lot of you have been frustrated in the past by not having your questions + contributions attended, but this is now changing as HuggingFace has stepped up to support the BNB project as an independent FOSS project. Looking forward to working on this together!

AMD integration
cc @fxmarty (HF) @arlo-phoenix @AloneLiberty @zoyamd @CorvetteCole @person4268 @luanrivello @Airradda @MatthewCroughan @devloic @ccbadd @IMbackK @Iron-Bound

Apple Silicon
cc @deep-pipeline @rickardp @eusip @phdykd @dosier @marcothedeveloper123 @ambahk @BalaajiSri @TheStoneMX

[deep-pipeline]

Just to say big thank you and to HF - delighted by this progress and appreciation of @rickardp & everyone's interest.
No need to cc: - though was appreciated! - heavily distracted by other things so likely intermittent look more than contributing. Good luck all!

[TheStoneMX]

I can help if you guys need help for cross-platform.

[Titus-von-Koeller]

@TheStoneMX Thanks a lot for offering to help ❤️ Do you have anything specific in mind? There are a bunch of topics that are related see here. Anything you see there that would spark your interest and match your expertise?

[Titus-von-Koeller]

Hey @wkpark,

I was just looking at the docs and one thing caught my eye: The requirement of Python 3.10 or newer in the Windows compilation requirements. In the workflow, 3.9 is the lowest supported version and generally the whole of BNB is 3.8 compatible.

Is there any reason why 3.8 wouldn't work for Windows support and, if no, should we also include 3.8 in the build matrix?

[akx]

@Titus-von-Koeller This was discussed in #1010 and #1011. Tests in CI would eventually run on all supported versions, but we should only need one wheel (per architecture/OS of course).

[matthewdouglas]

@Titus-von-Koeller I've opened #1076 to update the docs to indicate that Python 3.8 is the minimum. To @akx's point, I do think it makes sense for CI to eventually run tests on a wider matrix, but nothing special is needed for the build.

That said, Python 3.8 is going to be EOL in October and some dependencies are starting to drop 3.8 support in newer releases (e.g. scipy, numpy). I don't mind opening an RFC issue on dropping 3.8 if anyone's interested in that.

[BramVanroy]

I'm also interested to get this to work on HPC hardware (MIxxx series) but I am only a user so have no direct access to these machines nor the knowledge about rocm to help. Perhaps getting in touch with people at LUMI can be useful, since they have been working hard on improving the performance of AMD clusters (they are a full AMD supercluster).

[akx]

👍 I actually have development/R&D access to LUMI via my employer/startup/company @valohai 😊

I don't know anything about ROCm other than that it exists, though...

[Iron-Bound]

if you have MI250/300 cards, they may already be working on it here
https://github.com/ROCm/bitsandbytes/tree/rocm_enabled

[Titus-von-Koeller]

yeah, we're aware of that fork. It would be wonderful if we could somehow reunite these efforts with main line in the near mid-term. Thanks for the hint about LUMI @BramVanroy, might be interesting to engage with them.

[IMbackK]

This fork also works fine with MI25-MI100 with only some tiny modification to the makefile, not just mi200+ btw