Change the default model weights for HFTransformersNLP #7102

dakshvar22 · 2020-10-26T09:24:58Z

Proposed changes:

Closes https://github.com/RasaHQ/research/issues/119

Status (please check what you already did):

added some tests for the functionality
updated the documentation
updated the changelog (please check changelog for instructions)
reformat files using black (please check Readme for instructions)

dakshvar22 · 2020-10-26T09:34:49Z

Previous model regression tests available here. Running them again as those results were a week old, but expect them to not have changed.

github-actions · 2020-10-26T13:04:40Z

Commit: 49689ef, The full report is available as an artifact.

Dataset: Carbon Bot

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `1m19s`, train: `3m13s`, total: `4m32s`	0.7903 (0.02)	0.6260 (0.00)	0.4967 (`no data`)
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `1m35s`, train: `3m40s`, total: `5m15s`	0.7883 (0.01)	0.8199 (-0.04)	0.5648 (0.07)
`Sparse + BERT + DIET(bow) + ResponseSelector(bow)` test: `1m20s`, train: `3m30s`, total: `4m50s`	0.7903 (0.01)	0.6260 (0.00)	0.5232 (`no data`)
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `1m25s`, train: `4m4s`, total: `5m29s`	0.8039 (0.03)	0.8516 (-0.01)	0.5781 (0.03)

Dataset: Hermit

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `3m19s`, train: `20m24s`, total: `23m43s`	0.8931 (0.06)	0.7504 (0.00)	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m43s`, train: `12m27s`, total: `15m9s`	0.8894 (0.01)	0.8011 (0.01)	`no data`
`Sparse + BERT + DIET(bow) + ResponseSelector(bow)` test: `3m33s`, train: `23m43s`, total: `27m16s`	0.8662 (0.02)	0.7504 (0.00)	`no data`
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m51s`, train: `13m30s`, total: `16m21s`	0.8597 (-0.00)	0.7972 (0.01)	`no data`

Dataset: Private 1

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `1m54s`, train: `3m45s`, total: `5m39s`	0.8191 (0.00)	0.9612 (0.00)	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m0s`, train: `3m21s`, total: `5m20s`	0.8711 (0.00)	0.9680 (0.00)	`no data`

Dataset: Private 2

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `1m46s`, train: `3m37s`, total: `5m22s`	0.7256 (0.00)	`no data`	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `1m52s`, train: `7m13s`, total: `9m5s`	0.8371 (0.00)	`no data`	`no data`

Dataset: Private 3

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `49s`, train: `56s`, total: `1m45s`	0.4733 (0.00)	`no data`	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `51s`, train: `47s`, total: `1m38s`	0.5679 (0.00)	`no data`	`no data`

Dataset: Sara

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `2m8s`, train: `5m37s`, total: `7m45s`	0.8452 (0.07)	0.8683 (0.00)	0.8957 (0.11)
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m21s`, train: `4m13s`, total: `6m33s`	0.8570 (0.02)	0.8824 (-0.00)	0.8848 (0.09)
`Sparse + BERT + DIET(bow) + ResponseSelector(bow)` test: `2m13s`, train: `6m48s`, total: `9m1s`	0.8629 (0.02)	0.8683 (0.00)	0.8870 (0.00)
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m24s`, train: `4m50s`, total: `7m13s`	0.8717 (0.02)	0.8931 (-0.00)	0.8935 (0.01)

github-actions · 2020-10-26T13:35:43Z

Hey @dakshvar22! 👋 To run model regression tests, comment with the /modeltest command and a configuration.

Tips 💡: The model regression test will be run on push events. You can re-run the tests by re-add status:model-regression-tests label or use a Re-run jobs button in Github Actions workflow.

Tips 💡: Every time when you want to change a configuration you should edit the comment with the previous configuration.

You can copy this in your comment and customize:

/modeltest

```yml
##########
## Available datasets
##########
# - "Carbon Bot"
# - "Hermit"
# - "Private 1"
# - "Private 2"
# - "Private 3"
# - "Sara"

##########
## Available configurations
##########
# - "BERT + DIET(bow) + ResponseSelector(bow)"
# - "BERT + DIET(seq) + ResponseSelector(t2t)"
# - "Spacy + DIET(bow) + ResponseSelector(bow)"
# - "Spacy + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + BERT + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + BERT + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + Spacy + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)"

## Example configuration
#################### syntax #################
## include:
##   - dataset: ["<dataset_name>"]
##     config: ["<configuration_name>"]
#
## Example:
## include:
##  - dataset: ["Carbon Bot"]
##    config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]
#
## Shortcut:
## You can use the "all" shortcut to include all available configurations or datasets
#
## Example: Use the "Sparse + EmbeddingIntent + ResponseSelector(bow)" configuration
## for all available datasets
## include:
##  - dataset: ["all"]
##    config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]
#
## Example: Use all available configurations for the "Carbon Bot" and "Sara" datasets
## and for the "Hermit" dataset use the "Sparse + DIET + ResponseSelector(T2T)" and
## "BERT + DIET + ResponseSelector(T2T)" configurations:
## include:
##  - dataset: ["Carbon Bot", "Sara"]
##    config: ["all"]
##  - dataset: ["Hermit"]
##    config: ["Sparse + DIET(seq) + ResponseSelector(t2t)", "BERT + DIET(seq) + ResponseSelector(t2t)"]

include:
 - dataset: ["Carbon Bot"]
   config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]

```

github-actions · 2020-10-26T13:35:46Z

/modeltest

include:
 - dataset: ["Private 1", "Private 2", "Private 3"]
   config: ["BERT + DIET(bow) + ResponseSelector(bow)", "BERT + DIET(seq) + ResponseSelector(t2t)", "Sparse + BERT + DIET(bow) + ResponseSelector(bow)", "Sparse + BERT + DIET(seq) + ResponseSelector(t2t)"]

github-actions · 2020-10-26T13:35:48Z

The model regression tests have started. It might take a while, please be patient.
As soon as results are ready you'll see a new comment with the results.

Used configuration can be found in the comment.

tabergma

Results look good 💯

github-actions · 2020-10-26T14:27:27Z

Commit: b5068a2, The full report is available as an artifact.

Dataset: Private 1

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `1m49s`, train: `4m6s`, total: `5m55s`	0.9106 (0.09)	0.9612 (0.00)	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m13s`, train: `3m26s`, total: `5m38s`	0.9179 (0.05)	0.9699 (0.00)	`no data`

Dataset: Private 2

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `1m53s`, train: `3m45s`, total: `5m38s`	0.8757 (0.15)	`no data`	`no data`

Dataset: Private 3

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `53s`, train: `1m2s`, total: `1m55s`	0.9342 (0.46)	`no data`	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `56s`, train: `51s`, total: `1m47s`	0.8148 (0.25)	`no data`	`no data`

…nge_default_bert

change the default model name

3f2b723

dakshvar22 added the runner:gpu label Oct 26, 2020

add changelog

aefd118

dakshvar22 added the status:model-regression-tests label Oct 26, 2020

dakshvar22 added status:model-regression-tests and removed status:model-regression-tests labels Oct 26, 2020

dakshvar22 requested a review from tabergma October 26, 2020 09:35

github-actions bot deleted a comment from dakshvar22 Oct 26, 2020

dakshvar22 mentioned this pull request Oct 26, 2020

Bert experiments to validate the best layer for features #6967

Closed

4 tasks

github-actions bot removed status:model-regression-tests runner:gpu labels Oct 26, 2020

dakshvar22 added runner:gpu status:model-regression-tests labels Oct 26, 2020

Merge branch 'master' into change_default_bert

70f18cc

dakshvar22 added status:model-regression-tests and removed status:model-regression-tests labels Oct 26, 2020

tabergma approved these changes Oct 26, 2020

View reviewed changes

github-actions bot removed status:model-regression-tests runner:gpu labels Oct 26, 2020

dakshvar22 added the status:ready-to-merge label Oct 26, 2020

rasabot and others added 4 commits October 26, 2020 15:43

Merge branch 'master' into change_default_bert

cf37226

Merge branch 'master' into change_default_bert

5fad9d3

Merge branch 'master' into change_default_bert

1126d0f

change jobs to 1

58a687f

dakshvar22 removed the status:ready-to-merge label Oct 26, 2020

dakshvar22 added 7 commits October 26, 2020 19:01

fix tests add timeout

b84e3fa

change default models in tests

cfef1dc

change jobs 2

f2e0db9

Merge branch 'master' into change_default_bert

6623fe4

change how default params are passed to pipeline

c54dcb9

Merge branch 'change_default_bert' of github.com:RasaHQ/rasa into cha…

49e8368

…nge_default_bert

use bert-base-uncased everywhere in tests

5f22fd9

dakshvar22 added the status:ready-to-merge label Oct 27, 2020

Merge branch 'master' into change_default_bert

33a74b5

rasabot merged commit 1a39d06 into master Oct 27, 2020

rasabot deleted the change_default_bert branch October 27, 2020 10:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change the default model weights for HFTransformersNLP #7102

Change the default model weights for HFTransformersNLP #7102

dakshvar22 commented Oct 26, 2020 •

edited

dakshvar22 commented Oct 26, 2020

github-actions bot commented Oct 26, 2020

github-actions bot commented Oct 26, 2020

github-actions bot commented Oct 26, 2020

github-actions bot commented Oct 26, 2020

tabergma left a comment

github-actions bot commented Oct 26, 2020

Change the default model weights for HFTransformersNLP #7102

Change the default model weights for HFTransformersNLP #7102

Conversation

dakshvar22 commented Oct 26, 2020 • edited

dakshvar22 commented Oct 26, 2020

github-actions bot commented Oct 26, 2020

github-actions bot commented Oct 26, 2020

github-actions bot commented Oct 26, 2020

github-actions bot commented Oct 26, 2020

tabergma left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 26, 2020

dakshvar22 commented Oct 26, 2020 •

edited