Add Neuronx TGI #36
Add Neuronx TGI #36
Conversation
| @@ -0,0 +1,140 @@ | |||
| # Fetch and extract the TGI sources | |||
There was a problem hiding this comment.
Instead of including the sources for the tgi server in this repo, I would rather use those from the optimum-neuron repo that are likely to be up-to-date.
There was a problem hiding this comment.
You can cherry-pick the tip of my branch in my fork for instance:
https://github.com/dacorvo/llm-hosting-container/tree/neuronx-tgi
There was a problem hiding this comment.
for GPU we needed the dockerfile to be separate due to legal reviews.
There was a problem hiding this comment.
If you look at my branch I keep the Dockerfile, but I added a step to fetch the optimum-neuron sources and install from them directly.
There was a problem hiding this comment.
ah nice can you open a PR to my branch then we can keep this one?
There was a problem hiding this comment.
Note that I updated my branch to also remove the server files, so the commit i referenced before is invalid.
You can still git cherry-pick https://github.com/dacorvo/llm-hosting-container/tree/neuronx-tgi to get the correct commit.
There was a problem hiding this comment.
we can also close this PR and open a new one.
There was a problem hiding this comment.
I sent a pr to your branch as requested.
fix(tgi): install optimum-neuron and tgi server from original repo
|
Discarding in favor of #47. |
What does this PR?
This PR adds the Dockerfile and utiltilies to run TGI on AWS infernetia2. It also adds instruction on how to build the container. It includes more steps than for GPU since we need to build the optimum library first.