added Text Generation example #1473

Tegzes · 2022-02-25T23:25:24Z

Description

Please include a summary of the feature or issue being fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes #(1386)

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Feature/Issue validation/testing

Please describe the tests [UT/IT] that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

Test A
Test B
Logs

Checklist:

Have you added tests that prove your fix is effective or that this feature works?
Has code been commented, particularly in hard-to-understand areas?
Have you made corresponding changes to the documentation?

sagemaker-neo-ci-bot · 2022-02-25T23:48:38Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 7f7507f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-02-26T00:07:23Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 7f7507f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-02-26T00:14:33Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 7f7507f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

msaroufim

Looking good, very close - just needs a bit more polish

msaroufim · 2022-02-26T02:10:28Z

pull_request_template.md

@@ -9,9 +9,9 @@ Fixes #(issue)
 Please delete options that are not relevant.

 - [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
+- [x] New feature (non-breaking change which adds functionality)


Please revert the changes here, the template is automatically used when you open a new PR - no need to update it here

msaroufim · 2022-02-26T02:11:02Z

examples/Huggingface_Transformers/Transformer_handler_generalized.py

@@ -126,7 +129,7 @@ def preprocess(self, requests):
            max_length = self.setup_config["max_length"]
            logger.info("Received text: '%s'", input_text)
            # preprocessing text for sequence_classification and token_classification.
-            if self.setup_config["mode"] == "sequence_classification" or self.setup_config["mode"] == "token_classification":
+            if self.setup_config["mode"] == "sequence_classification" or self.setup_config["mode"] == "token_classification" or self.setup_config["mode"] == "text_generation":


nit: let's break this into multiple lines

maybe something like "if self.setup_config["mode"] in {"sequence_classification", "token_classification", "text_generation"}:" would be more elegant?

msaroufim · 2022-02-26T02:15:00Z

examples/Huggingface_Transformers/Transformer_handler_generalized.py

+                outputs = self.model.generate(input, max_length=150, do_sample=True, top_p=0.95, top_k=60)
+                generated = self.tokenizer.decode(input) + self.tokenizer.decode(outputs[0])[prompt_length + 1 :]
+
+                inferences.append(generated)


Just trying to understand this snippet. Are batches completely independent or are you chunking text from the same sentence? Should the batch size be equal to the max sequence length for this example to work?

Are the arguments the right for a demo? Is the model too slow? Is the generated output readable? Is the output too long? Is the output deterministic for your input? (important for tests and README) so people know they ran things correctly

outputs = self.model.generate(input, max_length=150, do_sample=True, top_p=0.95, top_k=60)

Also could you try printing the expected output?

I'd suggest adding a single test similar to this one https://github.com/pytorch/serve/blob/master/test/pytest/test_handler.py#L230 so it's easier for people to maintain this example without breaking it

Let me know if you need any more help with this. You put the mar file on any URL that makes it wgettable. I'm working on a cleaner longer frame story for this #1470

I tried to keep the same approach/style as the other transformer examples, so for each text example (prompt) from the batch, there is one instance of generated text (which is put together with the prompt, as in the official example). Max sequence is per every text example. The output will have the length of the max sequence and will be decoded in text. The parameters chosen in this function are the ones used in the official example from Huggingface (with a smaller max length), because I figured that maybe a lot of people would like see the same example served. Of course this might be improved in the future, by adding more flexibility in parameters selection.

The model size is comparable of the other transformers (~500 mb), but I'm not sure about the speed.

The output is not deterministic, because we can't know in advance what the generated text will be. From the local tests that I've done, each time I got a different output. I think the user can be sure the example is executed correctly when some text is generated, regardless of the content.

Considering this, I'm not sure how I should make assertion tests with this example.

Alright I think if that's the case please summarize the the first paragraph as a comment and update the README and we can merge this in

msaroufim · 2022-02-26T02:15:44Z

examples/Huggingface_Transformers/Download_Transformer_models.py

@@ -4,7 +4,7 @@
 import json
 import torch
 from transformers import (AutoModelForSequenceClassification, AutoTokenizer, AutoModelForQuestionAnswering,
- AutoModelForTokenClassification, AutoConfig)
+ AutoModelForTokenClassification, AutoModelForCausalLM, AutoConfig)


Make sure to update examples/Huggingface_Transformers/README.md as well

…n-example

sagemaker-neo-ci-bot · 2022-02-26T16:08:00Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: cc908cf
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-02-26T16:16:32Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: cc908cf
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-02-26T16:28:05Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: cc908cf
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-02-26T16:39:58Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: cd28007
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-02-26T17:00:13Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: cd28007
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-02-26T17:05:49Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: cd28007
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-02-26T21:01:32Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 9e279f4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-02-26T21:19:55Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 9e279f4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-02-26T21:26:35Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 9e279f4
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

msaroufim · 2022-02-26T23:05:04Z

examples/Huggingface_Transformers/Transformer_handler_generalized.py

+                outputs = self.model.generate(input, max_length=150, do_sample=True, top_p=0.95, top_k=60)
+                generated = self.tokenizer.decode(input) + self.tokenizer.decode(outputs[0])[prompt_length + 1 :]
+
+                inferences.append(generated)


Alright I think if that's the case please summarize the the first paragraph as a comment and update the README and we can merge this in

sagemaker-neo-ci-bot · 2022-02-27T10:28:09Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 7b46a6d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-02-27T10:35:22Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 7d8a1ee
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-02-27T10:43:26Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 7b46a6d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-02-27T10:51:05Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 7d8a1ee
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-02-27T10:51:20Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 7b46a6d
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-02-27T10:57:53Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 7d8a1ee
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

msaroufim

Excellent, thank you for your contribution!

msaroufim · 2022-03-01T02:13:15Z

Flagging @lxning @maaquib to merge after 0.5.3 release

sagemaker-neo-ci-bot · 2022-03-01T16:42:08Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 205082b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-03-01T17:00:10Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 205082b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-03-01T17:05:24Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 205082b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-03-03T18:15:56Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 42b59ac
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-03-03T18:31:04Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-cpu
Commit ID: 42b59ac
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-03-03T18:37:30Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 42b59ac
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-03-03T18:51:49Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-win
Commit ID: 42b59ac
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-neo-ci-bot · 2022-03-03T19:29:38Z

AWS CodeBuild CI Report

CodeBuild project: torch-serve-build-gpu
Commit ID: 42b59ac
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

added Text Generation example

7f7507f

msaroufim requested changes Feb 26, 2022

View reviewed changes

Tegzes added 2 commits February 26, 2022 16:44

print generated text

cc908cf

Merge branch 'master' of https://github.com/Tegzes/serve into text-ge…

96ccb23

…n-example

print generated text

cd28007

updated text generation example

9e279f4

Tegzes requested a review from msaroufim February 26, 2022 20:48

msaroufim requested changes Feb 26, 2022

View reviewed changes

Tegzes added 2 commits February 27, 2022 11:01

added comment for text gen example

7b46a6d

update readme for text generation example

7d8a1ee

Tegzes requested a review from msaroufim February 27, 2022 12:26

msaroufim approved these changes Feb 27, 2022

View reviewed changes

solved small print error

205082b

Merge branch 'master' into text-gen-example

42b59ac

maaquib approved these changes Mar 3, 2022

View reviewed changes

msaroufim merged commit 961eb59 into pytorch:master Mar 3, 2022

added Text Generation example #1473

added Text Generation example #1473

Conversation

Tegzes commented Feb 25, 2022

Description

Type of change

Feature/Issue validation/testing

Checklist:

sagemaker-neo-ci-bot commented Feb 25, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Feb 26, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Feb 26, 2022

AWS CodeBuild CI Report

msaroufim left a comment

Choose a reason for hiding this comment

msaroufim Feb 26, 2022

Choose a reason for hiding this comment

Tegzes Feb 26, 2022

Choose a reason for hiding this comment

msaroufim Feb 26, 2022

Choose a reason for hiding this comment

Tegzes Feb 26, 2022

Choose a reason for hiding this comment

msaroufim Feb 26, 2022 • edited

Choose a reason for hiding this comment

Tegzes Feb 26, 2022 • edited

Choose a reason for hiding this comment

msaroufim Feb 26, 2022

Choose a reason for hiding this comment

msaroufim Feb 26, 2022

Choose a reason for hiding this comment

sagemaker-neo-ci-bot commented Feb 26, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Feb 26, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Feb 26, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Feb 26, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Feb 26, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Feb 26, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Feb 26, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Feb 26, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Feb 26, 2022

AWS CodeBuild CI Report

msaroufim Feb 26, 2022

Choose a reason for hiding this comment

sagemaker-neo-ci-bot commented Feb 27, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Feb 27, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Feb 27, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Feb 27, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Feb 27, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Feb 27, 2022

AWS CodeBuild CI Report

msaroufim left a comment

Choose a reason for hiding this comment

msaroufim commented Mar 1, 2022

sagemaker-neo-ci-bot commented Mar 1, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Mar 1, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Mar 1, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Mar 3, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Mar 3, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Mar 3, 2022

AWS CodeBuild CI Report

sagemaker-neo-ci-bot commented Mar 3, 2022

msaroufim Feb 26, 2022 •

edited

Tegzes Feb 26, 2022 •

edited