Torchserve M1 support detailed plan

[msaroufim]

🚀 The feature

Update Jan 31 2024 - we should speed this up
PyTorch on MacOS x86 is being deprecated pytorch/pytorch#114602

Officially support torchserve on M1-M2 mac hardware. A bunch of people have asked for this, if you're interested please either upvote or volunteer to do this work, would be happy to review

We have 6 issues of people asking for M1 support m1

Motivation, pitch

I've seen a bunch of issues asking for this but we haven't managed to prioritize it yet so thought I'd highlight what needs to be done to support this. There's two components to this work, most of this work is parallelizable with the exception of the official release script so feel free to pick up an issue and work on it

Code changes

• In our base handler check if mps is available and if it is push the model and its inputs to an mps device for added hardware acceleration https://github.com/pytorch/serve/blob/master/ts/torch_handler/base_handler.py
• There will be many cases where mps might have some unsupported feature in which case it should be possible to also hardwire in config cpu fallback by putting in a flag in a model config like this one as an example https://github.com/pytorch/serve/blob/master/examples/large_models/inferentia2/model-config.yaml check out how this config is read and parsed in the base_handler.py
• Showcase an example of running llama2 at bs=1

Build changes

• Ensure ts_scripts/install_dependencies.py installs the right binaries for M1
• Ensure setup.py works on M1
• Ensure that conda binaries work with osx-arm64
• Change our CI job to run tests on M1 and deprecate the x86 version (Not deprecated yet simply added M1 compatibility)
• Ensure build scripts in binaries still work for M1 for both pip and conda
• In our official-release.yaml github action we should deprecate pushing regular mac binaries and instead move to using ARM M1 binaries
• Update our docker build scripts to also work with M1 Docker build on M1 fails #1363 (comment)

cc @agunapal @mreso @malfet

Alternatives

No response

Additional context

No response

[Talbot3]

did it ok?i am waiting ....

[udaij12]

Hi @Talbot3 What issues are you still facing?

[VladimirKadlec]

Hi, just FYI, ./build_image.sh builds a working docker image just fine on Macbook Air M2 running Asahi Linux (Ubuntu 23.10)

[Talbot3]

Hi @Talbot3 What issues are you still facing?

#1363
I just want check pytorch serve container on my device (mac m1) can run