Prompt Learning Inference Improvements #4566

vadam5 · 2022-07-19T22:45:28Z

Signed-off-by: Virginia Adams vadams@nvidia.com

Separates the gpt prompt learning eval script from the gpt model eval script to simplify inference script files and match t5 prompt learning inference workflow.

Collection: NLP

Changelog

Adds megatron_gpt_prompt_learning_eval.py script
Adds prompt learning inference config
Updated megatron_gpt_eval.py script
Updates gpt inference config
Updated gpt prompt learning dataset class
Updated gpt prompt learning model class generate and predict step methods

Usage

python megatron_gpt_prompt_learning_eval.py \
	virtual_prompt_model_file=models/squad_p_tuning_126m_tokens100_epochs100_layers4_1e-4.nemo \
        gpt_model_file=models/gpt_126m.nemo \
        inference.greedy=True \
        inference.add_BOS=False \
        trainer.devices=1 \
        tensor_model_parallel_size=1 \
        data_paths=["data/squad_short_test.jsonl"]

python megatron_gpt_prompt_learning_eval.py \
	virtual_prompt_model_file=models/squad_prompt_tuning_5b_tokens100_steps35000_2e-4.nemo \
        gpt_model_file=models/gpt_5b.nemo \
        inference.greedy=True \
        inference.add_BOS=False \
        trainer.devices=2 \
        tensor_model_parallel_size=2 \
        data_paths=["data/squad_short_test.jsonl"]

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: Virginia Adams <vadams@nvidia.com>

yidong72

some minor comments

yidong72 · 2022-07-20T12:48:23Z

examples/nlp/language_modeling/megatron_gpt_eval.py

+
+            for dataset in sentences:
+                for line in open(dataset, 'r', encoding='utf-8').readlines():
+                    self.sentences.append(line)


Here you still load all the sentences from the files into the memory. Why it avoids OOM?
Also consider add the doc string for RequestDataSet. the sentences can be multiple things now.

Ended up making a separate prompt learning eval script and removing the use of request dataset for prompt learning inference all together.

Previously, the inference code was trying to load and run inference on the entire test set at once without breaking it into batches. This was causing OOM for large test sets. This change was to make sure the data was loaded and passed to the model in batches. Though, like I said above, I ended up removing this part all together.

nemo/collections/nlp/data/language_modeling/megatron/gpt_prompt_learning_dataset.py

Signed-off-by: Virginia Adams <vadams@nvidia.com>

lgtm-com · 2022-07-20T22:51:21Z

This pull request introduces 2 alerts when merging 3482a32 into fea3775 - view on LGTM.com

new alerts:

2 for Unused import

lgtm-com · 2022-07-25T19:20:14Z

This pull request introduces 2 alerts when merging d74d42c into 6b9617d - view on LGTM.com

new alerts:

2 for Unused import

Signed-off-by: Virginia Adams <vadams@nvidia.com>

lgtm-com · 2022-07-25T20:05:59Z

This pull request introduces 2 alerts when merging a1f342c into 6b9617d - view on LGTM.com

new alerts:

2 for Unused import

Signed-off-by: Virginia Adams <vadams@nvidia.com>

lgtm-com · 2022-07-26T19:11:50Z

This pull request introduces 3 alerts when merging ccbe930 into 793cf48 - view on LGTM.com

new alerts:

3 for Unused import

lgtm-com · 2022-08-01T20:58:35Z

This pull request introduces 3 alerts when merging 150ceab into 1a9daa5 - view on LGTM.com

new alerts:

3 for Unused import

Signed-off-by: Virginia Adams <vadams@nvidia.com>

ericharper

LGTM. Thanks!

yidong72

some suggestion about api improvements

examples/nlp/language_modeling/megatron_gpt_prompt_learning_eval.py

nemo/collections/nlp/data/language_modeling/megatron/gpt_prompt_learning_dataset.py

Signed-off-by: Virginia Adams <vadams@nvidia.com>

yidong72

see my comment.

examples/nlp/language_modeling/megatron_gpt_prompt_learning_eval.py

Signed-off-by: Virginia Adams <vadams@nvidia.com>

lgtm-com · 2022-08-03T18:12:06Z

This pull request introduces 1 alert when merging 3f84478 into 08a623b - view on LGTM.com

new alerts:

1 for Wrong name for an argument in a class instantiation

Signed-off-by: Virginia Adams <vadams@nvidia.com>

yidong72

LGTM

Signed-off-by: Virginia Adams <vadams@nvidia.com>

* Improved prompt learning inference to avoid OOM Signed-off-by: Virginia Adams <vadams@nvidia.com> * Python style fix Signed-off-by: Virginia Adams <vadams@nvidia.com> * Made two prompt learning eval script seprate Signed-off-by: Virginia Adams <vadams@nvidia.com> * updated CI tests and documentation Signed-off-by: Virginia Adams <vadams@nvidia.com> * Removed request dataset from eval script Signed-off-by: Virginia Adams <vadams@nvidia.com> * Inference runs as expected Signed-off-by: Virginia Adams <vadams@nvidia.com> * address comments Signed-off-by: Virginia Adams <vadams@nvidia.com> * Changed input format for generate method Signed-off-by: Virginia Adams <vadams@nvidia.com> * Updated documentation, change dataset vairable name Signed-off-by: Virginia Adams <vadams@nvidia.com> * Updated unit tests Signed-off-by: Virginia Adams <vadams@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: David Mosallanezhad <dmosallanezh@nvidia.com>

* Improved prompt learning inference to avoid OOM Signed-off-by: Virginia Adams <vadams@nvidia.com> * Python style fix Signed-off-by: Virginia Adams <vadams@nvidia.com> * Made two prompt learning eval script seprate Signed-off-by: Virginia Adams <vadams@nvidia.com> * updated CI tests and documentation Signed-off-by: Virginia Adams <vadams@nvidia.com> * Removed request dataset from eval script Signed-off-by: Virginia Adams <vadams@nvidia.com> * Inference runs as expected Signed-off-by: Virginia Adams <vadams@nvidia.com> * address comments Signed-off-by: Virginia Adams <vadams@nvidia.com> * Changed input format for generate method Signed-off-by: Virginia Adams <vadams@nvidia.com> * Updated documentation, change dataset vairable name Signed-off-by: Virginia Adams <vadams@nvidia.com> * Updated unit tests Signed-off-by: Virginia Adams <vadams@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: George <zelenfr@ya.ru>

* Improved prompt learning inference to avoid OOM Signed-off-by: Virginia Adams <vadams@nvidia.com> * Python style fix Signed-off-by: Virginia Adams <vadams@nvidia.com> * Made two prompt learning eval script seprate Signed-off-by: Virginia Adams <vadams@nvidia.com> * updated CI tests and documentation Signed-off-by: Virginia Adams <vadams@nvidia.com> * Removed request dataset from eval script Signed-off-by: Virginia Adams <vadams@nvidia.com> * Inference runs as expected Signed-off-by: Virginia Adams <vadams@nvidia.com> * address comments Signed-off-by: Virginia Adams <vadams@nvidia.com> * Changed input format for generate method Signed-off-by: Virginia Adams <vadams@nvidia.com> * Updated documentation, change dataset vairable name Signed-off-by: Virginia Adams <vadams@nvidia.com> * Updated unit tests Signed-off-by: Virginia Adams <vadams@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: Anas Abou Allaban <aabouallaban@pm.me>

* Improved prompt learning inference to avoid OOM Signed-off-by: Virginia Adams <vadams@nvidia.com> * Python style fix Signed-off-by: Virginia Adams <vadams@nvidia.com> * Made two prompt learning eval script seprate Signed-off-by: Virginia Adams <vadams@nvidia.com> * updated CI tests and documentation Signed-off-by: Virginia Adams <vadams@nvidia.com> * Removed request dataset from eval script Signed-off-by: Virginia Adams <vadams@nvidia.com> * Inference runs as expected Signed-off-by: Virginia Adams <vadams@nvidia.com> * address comments Signed-off-by: Virginia Adams <vadams@nvidia.com> * Changed input format for generate method Signed-off-by: Virginia Adams <vadams@nvidia.com> * Updated documentation, change dataset vairable name Signed-off-by: Virginia Adams <vadams@nvidia.com> * Updated unit tests Signed-off-by: Virginia Adams <vadams@nvidia.com> Co-authored-by: Eric Harper <complex451@gmail.com> Signed-off-by: Hainan Xu <hainanx@nvidia.com>

Improved prompt learning inference to avoid OOM

46da5dd

Signed-off-by: Virginia Adams <vadams@nvidia.com>

vadam5 requested review from yidong72 and ericharper July 19, 2022 22:45

Python style fix

edba0e8

Signed-off-by: Virginia Adams <vadams@nvidia.com>

yidong72 reviewed Jul 20, 2022

View reviewed changes

Made two prompt learning eval script seprate

3482a32

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Merge branch 'main' into prompt_learning_inference_improvements

d74d42c

updated CI tests and documentation

a1f342c

Signed-off-by: Virginia Adams <vadams@nvidia.com>

vadam5 changed the title ~~Improved prompt learning inference to avoid OOM~~ Prompt Learning Inference Improvements Jul 26, 2022

Removed request dataset from eval script

ccbe930

Signed-off-by: Virginia Adams <vadams@nvidia.com>

Merge branch 'main' into prompt_learning_inference_improvements

150ceab

Inference runs as expected

31acc60

Signed-off-by: Virginia Adams <vadams@nvidia.com>

vadam5 requested a review from yidong72 August 1, 2022 21:59

ericharper previously approved these changes Aug 2, 2022

View reviewed changes

Merge branch 'main' into prompt_learning_inference_improvements

dc4df73

yidong72 reviewed Aug 2, 2022

View reviewed changes

examples/nlp/language_modeling/megatron_gpt_prompt_learning_eval.py Outdated Show resolved Hide resolved

nemo/collections/nlp/data/language_modeling/megatron/gpt_prompt_learning_dataset.py Outdated Show resolved Hide resolved

address comments

5dae71f

Signed-off-by: Virginia Adams <vadams@nvidia.com>

vadam5 dismissed ericharper’s stale review via 5dae71f August 3, 2022 00:01

Merge branch 'main' into prompt_learning_inference_improvements

f081a6d

vadam5 requested a review from yidong72 August 3, 2022 00:01

yidong72 reviewed Aug 3, 2022

View reviewed changes

examples/nlp/language_modeling/megatron_gpt_prompt_learning_eval.py Outdated Show resolved Hide resolved

Changed input format for generate method

3f84478

Signed-off-by: Virginia Adams <vadams@nvidia.com>

vadam5 requested a review from yidong72 August 3, 2022 18:07

Updated documentation, change dataset vairable name

e3a1ff7

Signed-off-by: Virginia Adams <vadams@nvidia.com>

yidong72 previously approved these changes Aug 3, 2022

View reviewed changes

Updated unit tests

0015b56

Signed-off-by: Virginia Adams <vadams@nvidia.com>

vadam5 dismissed yidong72’s stale review via 0015b56 August 3, 2022 21:27

Merge branch 'main' into prompt_learning_inference_improvements

22e0c5d

yidong72 approved these changes Aug 3, 2022

View reviewed changes

vadam5 added 2 commits August 3, 2022 15:14

Merge branch 'main' into prompt_learning_inference_improvements

27b9fdb

Merge branch 'main' into prompt_learning_inference_improvements

6d04021

vadam5 merged commit f39bc66 into main Aug 4, 2022

vadam5 mentioned this pull request Aug 9, 2022

Typo #4554

Closed

XuesongYang mentioned this pull request Aug 10, 2022

Fix typo (cartographic -> catastrophic) #4691

Closed

8 tasks

ericharper deleted the prompt_learning_inference_improvements branch September 20, 2022 23:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prompt Learning Inference Improvements #4566

Prompt Learning Inference Improvements #4566

vadam5 commented Jul 19, 2022 •

edited

Loading

yidong72 left a comment

yidong72 Jul 20, 2022

vadam5 Jul 25, 2022 •

edited

Loading

lgtm-com bot commented Jul 20, 2022

lgtm-com bot commented Jul 25, 2022

lgtm-com bot commented Jul 25, 2022

lgtm-com bot commented Jul 26, 2022

lgtm-com bot commented Aug 1, 2022

ericharper left a comment

yidong72 left a comment

yidong72 left a comment

lgtm-com bot commented Aug 3, 2022

yidong72 left a comment

Prompt Learning Inference Improvements #4566

Prompt Learning Inference Improvements #4566

Conversation

vadam5 commented Jul 19, 2022 • edited Loading

Changelog

Usage

Before your PR is "Ready for review"

Who can review?

Additional Information

yidong72 left a comment

Choose a reason for hiding this comment

yidong72 Jul 20, 2022

Choose a reason for hiding this comment

vadam5 Jul 25, 2022 • edited Loading

Choose a reason for hiding this comment

lgtm-com bot commented Jul 20, 2022

lgtm-com bot commented Jul 25, 2022

lgtm-com bot commented Jul 25, 2022

lgtm-com bot commented Jul 26, 2022

lgtm-com bot commented Aug 1, 2022

ericharper left a comment

Choose a reason for hiding this comment

yidong72 left a comment

Choose a reason for hiding this comment

yidong72 left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Aug 3, 2022

yidong72 left a comment

Choose a reason for hiding this comment

vadam5 commented Jul 19, 2022 •

edited

Loading

vadam5 Jul 25, 2022 •

edited

Loading