Include two different stages for building TGI image: #34
Conversation
mfuntowicz
commented
May 2, 2024
mfuntowicz
commented
- Default standalone image
- Inference Endpoint specific image
- Default standalone image - Inference Endpoint specific image
| FROM tpu_base as final_image | ||
| ENTRYPOINT text-generation-launcher | ||
| CMD ["--json-output"] |
There was a problem hiding this comment.
this seems to revert 8756c8a, are you sure about it?
There was a problem hiding this comment.
For the case of non inference-endpoints (IE) Docker image we want the image to provide their own argument, so this was correct and the changes we did were too focus on IE loosing flexibility in other deployment scenarios
There was a problem hiding this comment.
Ok, in that case we might want to keep 2 docker images, one for TGI non inference-endpoints (IE), the other for IE. Or, if we are not interested in maintaining one of those, we should probably drop the image and the associated targets.
There was a problem hiding this comment.
Yes but we are definitely interested in keeping both and potentially adding a third one for vertex if it makes sense later 👍🏻
| --build-arg VERSION=$(VERSION) \ | ||
| --build-arg TGI_VERSION=$(TGI_VERSION) \ | ||
| -t huggingface/optimum-tpu:$(VERSION)-tgi . | ||
| docker tag huggingface/optimum-tpu:$(VERSION)-tgi huggingface/optimum-tpu:latest | ||
|
|
||
| tpu-tgi-ie: |
There was a problem hiding this comment.
target tgi_docker_test should depend on this (and we should probably fix it at some point)
There was a problem hiding this comment.
Does the tgi_docker_test needs to be run on an inference-endpoints specific docker image?
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
