Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement metal kernel for GPUs on Mac #44

Open
casper-hansen opened this issue Sep 11, 2023 · 0 comments
Open

Implement metal kernel for GPUs on Mac #44

casper-hansen opened this issue Sep 11, 2023 · 0 comments
Labels
help wanted Extra attention is needed

Comments

@casper-hansen
Copy link
Owner

casper-hansen commented Sep 11, 2023

MIT released their TinyChatEngine which includes kernels for every kind of platform. To ensure wide availability, we should integrate the kernels for Metal which are kernels for running quantized on MacOS.

Requirements:

  • Run the Metal kernel through Python (maybe run on the fly?)
  • Automatically switch to the Metal kernel if torch.backends.mps.is_available(), otherwise use CUDA.
  • Figure out: How to support either GEMV/GEMM format with Metal

Kernel:
https://github.com/mit-han-lab/TinyChatEngine/blob/main/kernels/metal/kernel/op.metal#L10

@casper-hansen casper-hansen added the help wanted Extra attention is needed label Sep 11, 2023
@casper-hansen casper-hansen mentioned this issue Sep 11, 2023
30 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

1 participant