-
Notifications
You must be signed in to change notification settings - Fork 501
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[EKS] Support Inferentia/Neuron Runtime #1995
Comments
Thanks for raising this. We're interested in integrating with Neuron, and it's something we're planning to look into down the road! |
Re-titled this to be consistent with #1075, which is similar but for an ECS Inferentia variant. |
Is this still needed? |
Yes, we are using more neuron instances than I created the ticket. (actively migrating workloads from gpu to neuron) |
Container SSA check-in. IHAC is running ML workloads with Inferentia on EKS. They are quite interested in Bottlerocket in terms of awesome security benefits they get with less overhead. They really want to align the company standards to use Bottlerocket for general business application as well as ML workloads. But the lack of support for Inferentia would affect their adoption. |
IHAC who is running Stable Diffusion on EKS Inf2, and they wish to adopt Bottlerocket image cache solution to reduce the large image (10+GB) pulling time from ECR around 3-4 minutes. Foreseeing the increasing GenAI model hosting with Inferentia, supporting Inferentia/Neuron runtime will have a big impact. |
What I'd like:
I think it requires the neuron driver on https://github.com/aws/aws-neuron-sdk
Any alternatives you've considered:
Nothing
The text was updated successfully, but these errors were encountered: