Add slot filling into DST Generative model #3695

Zhilin123 · 2022-02-16T23:33:49Z

What does this PR do ?

See Design Doc and Milestones for Overarching project at Nvidia-only access Google Docs

Add slot filling prediction capabilities and metrics

Collection: NLP

Changelog

Integrated the capability of performing intent classification and slot filling jointly using a GPT-style model
Include metrics for slot filling
Edited dataset class to improve user customizability in using templates for generation
Backported IntentSlotClassificationModel into DST module

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

…/NVIDIA/NeMo into main

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

…/NVIDIA/NeMo into main

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

…/NVIDIA/NeMo into main

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

…/NVIDIA/NeMo into main

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

…_init__.py Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

lgtm-com · 2022-03-03T03:28:55Z

This pull request introduces 3 alerts and fixes 1 when merging f929efc into 8b5e2e8 - view on LGTM.com

new alerts:

3 for Unused import

fixed alerts:

1 for Unused import

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

…/NVIDIA/NeMo into main

lgtm-com · 2022-03-07T06:11:29Z

This pull request introduces 1 alert and fixes 1 when merging 070596d into 69385da - view on LGTM.com

new alerts:

1 for Unused import

fixed alerts:

1 for Unused import

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

lgtm-com · 2022-03-08T07:18:16Z

This pull request introduces 1 alert and fixes 1 when merging 4c3f188 into 989e6ab - view on LGTM.com

new alerts:

1 for Unused import

fixed alerts:

1 for Unused import

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

lgtm-com · 2022-03-10T21:56:18Z

This pull request fixes 1 alert when merging 17712d0 into ec99f42 - view on LGTM.com

fixed alerts:

1 for Unused import

vadam5

Left some comments and questions.

vadam5 · 2022-03-10T22:11:07Z

nemo/collections/nlp/data/dialogue_state_tracking_generative/sgd/dialogue_bert_dataset.py

-        raise NotImplementedError()
+
+    """
+    Creates dataset to use for the task of joint intent


Just some grammatical edits "Creates a dataset to use for the task of joint intent and slot classification with a pretrained model"

vadam5 · 2022-03-10T22:11:35Z

nemo/collections/nlp/data/dialogue_state_tracking_generative/sgd/dialogue_bert_dataset.py

+    Converts from raw data to an instance that can be used by
+    NMDataLayer.
+
+    For dataset to use during inference without labels, see


"For a dataset to use"

examples/nlp/dialogue_state_tracking_generative/sgd_gen.py

nemo/collections/nlp/data/dialogue_state_tracking_generative/sgd/dialogue_bert_dataset.py

nemo/collections/nlp/models/dialogue_state_tracking_generative/dialogue_gpt_model.py

vadam5 · 2022-03-11T20:28:43Z

nemo/collections/nlp/models/dialogue_state_tracking_generative/dialogue_gpt_model.py

+            # 'assit_intent_and_slot_with_options' has prompt_id of 2
+            prompt_ids = torch.tensor([1] * input_ids.size(0)) if self.prompt_tags else None
+            if self.cfg.dataset.prompt_template == "prompt_tuning_with_options" and prompt_ids is not None:
+                prompt_ids = prompt_ids * 2


Why do you double the prompt ID list?

nemo/collections/nlp/models/dialogue_state_tracking_generative/dialogue_gpt_model.py

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

lgtm-com · 2022-03-11T23:10:40Z

This pull request fixes 1 alert when merging 383b5bd into 3dd8a5c - view on LGTM.com

fixed alerts:

1 for Unused import

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

lgtm-com · 2022-03-12T00:31:17Z

This pull request fixes 1 alert when merging 8dc880a into 814be88 - view on LGTM.com

fixed alerts:

1 for Unused import

vadam5

LGTM

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

lgtm-com · 2022-03-12T00:57:18Z

This pull request fixes 1 alert when merging c4408c6 into 814be88 - view on LGTM.com

fixed alerts:

1 for Unused import

lgtm-com · 2022-03-14T00:03:44Z

This pull request fixes 1 alert when merging b2724c4 into b31d8aa - view on LGTM.com

fixed alerts:

1 for Unused import

lgtm-com · 2022-03-14T16:17:38Z

This pull request fixes 1 alert when merging e073993 into 1b28c8c - view on LGTM.com

fixed alerts:

1 for Unused import

* refactor dialogue state tracking for modelling/dataset interoperability Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * fix style changes Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * fix typo Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * fix style raised by lgtm Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * fix style formatting Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * update template to include description of intent Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * update Jenkinsfile Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * changes based on requests in review Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * add compatibility with assistant dataset Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * update Jenkins Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * remove dialogue_state_tracking Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * update huggingface utils for dialogue Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * rename dialogue_state_tracking_hybrid to dialogue_state_tracking_sgdqa Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * style fix Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * fix style Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * style fix nemo/collections/nlp/models/dialogue_state_tracking_sgdqa/__init__.py Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * update Jenkinsfile for SGDGEN Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * update Jenkinsfile for SGDGEN Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * update Jenkinsfile for SGDGEN Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * update Jenkinsfile for SGDGEN Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * update Jenkinsfile for SGDGEN Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * fix typo Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * add docstrings for assistant data processsor Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * update Jenkins for SGDGEN local checkpoint Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * update style Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * use local vocab file for Jenkinsfile Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * patch for Jenkins CI using local file Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * add slot filling prediction and metrics Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * remove unused code Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * style fix Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * refactor metrics code out of Dialogue GPT Model Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * integrate backward compatible support for IntentSlotClassificationModel (bert model) Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * save prediction file for IntentSlotClassification Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * update dialogue gpt model training for megatron gpt Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * remove batch generate for HF GPT2, which causes lower performance Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * add few shot capability to dialogue gpt model Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * update Jenkinsfile and remove unused import Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * update code description and clarity Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * address PR comments Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> * style fix Signed-off-by: Zhilin Wang <zhilinw@nvidia.com> Co-authored-by: Zhilin Wang <zhilinw@nvidia.com> Co-authored-by: Oleksii Kuchaiev <okuchaiev@users.noreply.github.com> Co-authored-by: Yang Zhang <yzhang123@users.noreply.github.com> Co-authored-by: Eric Harper <complex451@gmail.com>

Zhilin123 and others added 30 commits January 26, 2022 14:45

refactor dialogue state tracking for modelling/dataset interoperability

b287422

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

fix style changes

39dbec3

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

Merge branch 'main' into dialogue_state_tracking_refactor

4bd1cc1

Merge branch 'dialogue_state_tracking_refactor' of https://github.com…

7c12ab6

…/NVIDIA/NeMo into main

fix typo

23f33f9

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

fix style raised by lgtm

255294d

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

fix style formatting

5f6dbd9

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

Merge branch 'main' into dialogue_state_tracking_refactor

deeeaec

update template to include description of intent

4e08423

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

Merge branch 'main' into dialogue_state_tracking_refactor

7d96cf2

Merge branch 'dialogue_state_tracking_refactor' of https://github.com…

0b1bc6c

…/NVIDIA/NeMo into main

Merge branch 'main' into dialogue_state_tracking_refactor

f5f3cf8

update Jenkinsfile

e1cbfe8

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

Merge branch 'dialogue_state_tracking_refactor' of https://github.com…

6a80dd4

…/NVIDIA/NeMo into main

Merge branch 'main' into dialogue_state_tracking_refactor

d5e204f

changes based on requests in review

87fd90f

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

Merge branch 'dialogue_state_tracking_refactor' of https://github.com…

f28ac4a

…/NVIDIA/NeMo into main

add compatibility with assistant dataset

ce5e3bc

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

update Jenkins

600abe5

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

remove dialogue_state_tracking

b5f71c8

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

update huggingface utils for dialogue

60a907f

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

rename dialogue_state_tracking_hybrid to dialogue_state_tracking_sgdqa

d76d5e9

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

style fix

0fdec50

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

fix style

b665b10

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

style fix nemo/collections/nlp/models/dialogue_state_tracking_sgdqa/_…

67ab0d2

…_init__.py Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

update Jenkinsfile for SGDGEN

e4418ce

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

update Jenkinsfile for SGDGEN

012004d

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

update Jenkinsfile for SGDGEN

8241502

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

update Jenkinsfile for SGDGEN

1a1bc0d

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

update Jenkinsfile for SGDGEN

f7a7e9c

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

Zhilin123 added 2 commits March 6, 2022 21:57

remove batch generate for HF GPT2, which causes lower performance

dccb01a

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

Merge branch 'dialogue_state_tracking_refactor' of https://github.com…

070596d

…/NVIDIA/NeMo into main

add few shot capability to dialogue gpt model

4c3f188

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

Zhilin123 and others added 2 commits March 10, 2022 13:39

update Jenkinsfile and remove unused import

c20057a

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

Merge branch 'main' into dialogue_state_tracking_refactor

17712d0

vadam5 suggested changes Mar 11, 2022

View reviewed changes

Zhilin123 and others added 2 commits March 11, 2022 14:59

update code description and clarity

d0d995f

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

Merge branch 'main' into dialogue_state_tracking_refactor

383b5bd

vadam5 self-requested a review March 11, 2022 23:08

Zhilin123 and others added 2 commits March 11, 2022 16:14

address PR comments

f206d8f

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

Merge branch 'main' into dialogue_state_tracking_refactor

8dc880a

vadam5 previously approved these changes Mar 12, 2022

View reviewed changes

style fix

c4408c6

Signed-off-by: Zhilin Wang <zhilinw@nvidia.com>

Zhilin123 dismissed vadam5’s stale review via c4408c6 March 12, 2022 00:46

vadam5 self-requested a review March 12, 2022 00:47

vadam5 approved these changes Mar 12, 2022

View reviewed changes

Merge branch 'main' into dialogue_state_tracking_refactor

b2724c4

Merge branch 'main' into dialogue_state_tracking_refactor

e073993

Zhilin123 merged commit fe48fa7 into main Mar 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add slot filling into DST Generative model #3695

Add slot filling into DST Generative model #3695

Zhilin123 commented Feb 16, 2022 •

edited

Loading

lgtm-com bot commented Mar 3, 2022

lgtm-com bot commented Mar 7, 2022

lgtm-com bot commented Mar 8, 2022

lgtm-com bot commented Mar 10, 2022

vadam5 left a comment

vadam5 Mar 10, 2022

vadam5 Mar 10, 2022

vadam5 Mar 11, 2022

lgtm-com bot commented Mar 11, 2022

lgtm-com bot commented Mar 12, 2022

vadam5 left a comment

lgtm-com bot commented Mar 12, 2022

lgtm-com bot commented Mar 14, 2022

lgtm-com bot commented Mar 14, 2022

Add slot filling into DST Generative model #3695

Add slot filling into DST Generative model #3695

Conversation

Zhilin123 commented Feb 16, 2022 • edited Loading

What does this PR do ?

Changelog

Usage

Before your PR is "Ready for review"

Who can review?

Additional Information

lgtm-com bot commented Mar 3, 2022

lgtm-com bot commented Mar 7, 2022

lgtm-com bot commented Mar 8, 2022

lgtm-com bot commented Mar 10, 2022

vadam5 left a comment

Choose a reason for hiding this comment

vadam5 Mar 10, 2022

Choose a reason for hiding this comment

vadam5 Mar 10, 2022

Choose a reason for hiding this comment

vadam5 Mar 11, 2022

Choose a reason for hiding this comment

lgtm-com bot commented Mar 11, 2022

lgtm-com bot commented Mar 12, 2022

vadam5 left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Mar 12, 2022

lgtm-com bot commented Mar 14, 2022

lgtm-com bot commented Mar 14, 2022

Zhilin123 commented Feb 16, 2022 •

edited

Loading