-
Notifications
You must be signed in to change notification settings - Fork 832
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TorchServe quick start example #3040
base: master
Are you sure you want to change the base?
Conversation
This example contains Dockerfile and the config+handler for BERT, Resnet, and Desnet. It's not clear for cx to follow up this example since TS has dir for Dockerfile and model handlers for these models. We need have a clear guidance and implementation to support model card. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
personally I like the direction this is going in. Recommending docker early on to users feels like a natural thing since torchserve will just work on any device
I also like making the compiler flag an env variable will make trying things out quite simple
What I don't like is that bert examples still have these unsinspiring labels for "accepted" vs "not accepted" which is confusing because it might make people think it's a sytem level error instead of the expected output of the mode. I would also want to see the code duplication issues fixed. There's also still a link to an existing getting started guide on the main page, let's get rid of that and the tutorial
Otherwise once this is in I do expect this to be default way people first use torchserve, presumably you're doing llama7b and 70b in a future PR?
@@ -0,0 +1,4 @@ | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
these labels have been confusing forever, can we do something else?
@@ -0,0 +1,538 @@ | |||
import ast |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this copy pasted from the examples transformer handler? The duplication will hurt us would recommend having this in 1 place
@@ -0,0 +1,188 @@ | |||
import argparse |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same point on duplication here
COPY $EXAMPLE_DIR/config.properties /home/model-server/config.properties | ||
|
||
WORKDIR /home/model-server/getting_started | ||
RUN chmod +x /usr/local/bin/dockerd-entrypoint.sh \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cool!
Description
The goal for the PR is to enable TorchServe user to quickly deploy a model for serving using a single script without having to install anything manually.
This PR adds the following quick start example for TorchServe
torch.compile
Fixes #(issue)
Type of change
Please delete options that are not relevant.
Feature/Issue validation/testing
Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.
Checklist: