Remove reg grad filtering to see performance diff #9867

dakshvar22 · 2021-10-13T19:01:48Z

Proposed changes:

Check the effect of reverting Filter grads #5669

Status (please check what you already did):

added some tests for the functionality
updated the documentation
updated the changelog (please check changelog for instructions)
reformat files using black (please check Readme for instructions)

etry install --extras full

github-actions · 2021-10-18T21:40:54Z

Commit: 9f15a18, The full report is available as an artifact.

github-actions · 2021-10-22T10:52:52Z

Commit: bba1d43, The full report is available as an artifact.

Dataset: Carbon Bot, Dataset repository branch: main, commit: 15f3a3ed0e8f9d592c47b3f6e5f927eb5b891854

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `1m45s`, train: `4m0s`, total: `5m44s`	0.7495 (0.01)	0.7049 (0.00)	0.5099 (-0.01)

github-actions · 2021-10-22T11:54:44Z

Commit: bba1d43, The full report is available as an artifact.

Dataset: Carbon Bot, Dataset repository branch: main, commit: 15f3a3ed0e8f9d592c47b3f6e5f927eb5b891854

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `3m12s`, train: `15m27s`, total: `18m39s`	0.7922 (0.01)	0.8032 (0.02)	0.5430 (-0.01)
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `2m1s`, train: `15m45s`, total: `17m46s`	0.7495 (0.01)	0.7049 (0.00)	0.5298 (0.01)

github-actions · 2021-11-17T10:44:39Z

Commit: 259f60d, The full report is available as an artifact.

Dataset: Carbon Bot, Dataset repository branch: main, commit: 819cb7b3cc077753e67178ad022d577f164e99cf

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m46s`, train: `5m57s`, total: `8m42s`	0.7922 (-0.02)	0.7810 (0.00)	0.5467 (0.02)
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m55s`, train: `5m52s`, total: `8m46s`	0.7922 (0.01)	0.8032 (0.02)	0.5232 (-0.06)
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `1m49s`, train: `4m17s`, total: `6m5s`	0.7495 (0.01)	0.7049 (0.00)	0.5033 (-0.03)

Dataset: financial-demo, Dataset repository branch: fix-model-regression-tests (external repository), commit: 52a3ad3eb5292d56542687e23b06703431f15ead
Configuration repository branch: main

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `35s`, train: `52s`, total: `1m26s`	1.0000 (0.00)	0.8333 (0.00)	`no data`
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `34s`, train: `54s`, total: `1m28s`	1.0000 (0.00)	0.8800 (0.00)	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `27s`, train: `47s`, total: `1m13s`	0.9643 (0.00)	0.8800 (0.00)	`no data`

github-actions · 2021-11-17T12:27:13Z

Commit: 259f60d, The full report is available as an artifact.

Dataset: Carbon Bot, Dataset repository branch: main, commit: 819cb7b3cc077753e67178ad022d577f164e99cf

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m47s`, train: `4m55s`, total: `7m41s`	0.7922 (-0.02)	0.7810 (0.00)	0.5467 (0.02)
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m54s`, train: `4m52s`, total: `7m46s`	0.7922 (0.01)	0.8032 (0.02)	0.5762 (-0.01)
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `1m49s`, train: `4m11s`, total: `6m0s`	0.7495 (0.01)	0.7049 (0.00)	0.5232 (-0.02)

Dataset: Private 1, Dataset repository branch: main, commit: 819cb7b3cc077753e67178ad022d577f164e99cf

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `3m45s`, train: `3m17s`, total: `7m2s`	0.9158 (0.00)	0.9691 (-0.00)	`no data`
`Spacy + DIET(seq) + ResponseSelector(t2t)` test: `2m10s`, train: `3m19s`, total: `5m29s`	0.8565 (0.00)	0.9373 (-0.01)	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `1m58s`, train: `3m5s`, total: `5m3s`	0.9023 (-0.00)	0.9725 (0.00)	`no data`
`Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)` test: `2m15s`, train: `3m39s`, total: `5m53s`	0.8846 (-0.01)	0.9649 (-0.00)	`no data`

Dataset: financial-demo, Dataset repository branch: fix-model-regression-tests (external repository), commit: 52a3ad3eb5292d56542687e23b06703431f15ead
Configuration repository branch: main

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `35s`, train: `51s`, total: `1m26s`	1.0000 (0.00)	0.8333 (0.00)	`no data`
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `37s`, train: `55s`, total: `1m31s`	1.0000 (0.00)	0.8800 (0.00)	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `28s`, train: `45s`, total: `1m12s`	0.9643 (0.00)	0.8800 (0.00)	`no data`

github-actions · 2021-11-23T12:08:32Z

Commit: 852c021, The full report is available as an artifact.

Dataset: Carbon Bot, Dataset repository branch: main, commit: 819cb7b3cc077753e67178ad022d577f164e99cf

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `4m35s`, train: `19m32s`, total: `24m7s`	0.7922 (-0.02)	0.7810 (0.00)	0.5467 (0.02)
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `3m4s`, train: `14m17s`, total: `17m20s`	0.7922 (0.01)	0.8032 (0.02)	0.5430 (-0.04)
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `1m24s`, train: `10m18s`, total: `11m41s`	0.7495 (0.01)	0.7049 (0.00)	0.5232 (-0.02)

Dataset: Private 1, Dataset repository branch: main, commit: 819cb7b3cc077753e67178ad022d577f164e99cf

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `4m54s`, train: `11m51s`, total: `16m44s`	0.9158 (0.00)	0.9691 (-0.00)	`no data`
`Spacy + DIET(seq) + ResponseSelector(t2t)` test: `1m59s`, train: `10m34s`, total: `12m33s`	0.8565 (0.00)	0.9373 (-0.01)	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `1m27s`, train: `9m0s`, total: `10m27s`	0.9002 (-0.00)	0.9744 (0.01)	`no data`
`Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)` test: `1m38s`, train: `9m21s`, total: `10m59s`	0.8846 (-0.01)	0.9668 (-0.00)	`no data`

Dataset: financial-demo, Dataset repository branch: fix-model-regression-tests (external repository), commit: 52a3ad3eb5292d56542687e23b06703431f15ead
Configuration repository branch: main

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `1m9s`, train: `3m10s`, total: `4m19s`	1.0000 (0.00)	0.8333 (0.00)	`no data`
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `1m17s`, train: `2m46s`, total: `4m4s`	1.0000 (0.00)	0.8800 (0.00)	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `25s`, train: `1m23s`, total: `1m47s`	0.9643 (0.00)	0.8800 (0.00)	`no data`

github-actions · 2021-11-23T15:11:16Z

Hey @markus-hinsche! 👋 To run model regression tests, comment with the /modeltest command and a configuration.

Tips 💡: The model regression test will be run on push events. You can re-run the tests by re-add status:model-regression-tests label or use a Re-run jobs button in Github Actions workflow.

Tips 💡: Every time when you want to change a configuration you should edit the comment with the previous configuration.

You can copy this in your comment and customize:

/modeltest

```yml
##########
## Available datasets
##########
# - "Carbon Bot" (NLU)
# - "Hermit" (NLU)
# - "Private 1" (NLU)
# - "Private 2" (NLU)
# - "Private 3" (NLU)
# - "Sara" (NLU, Core)
# - "financial-demo" (NLU, Core)
# - "helpdesk-assistant" (NLU, Core)
# - "insurance-demo" (NLU, Core)
# - "retail-demo" (NLU, Core)

##########
## Available NLU configurations
##########
# - "BERT + DIET(bow) + ResponseSelector(bow)"
# - "BERT + DIET(seq) + ResponseSelector(t2t)"
# - "Spacy + DIET(bow) + ResponseSelector(bow)"
# - "Spacy + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + BERT + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + BERT + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + Spacy + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)"

##########
## Available Core configurations
##########
# - "Rules"
# - "Rules + AugMemo"
# - "Rules + AugMemo + TED"
# - "Rules + Memo"
# - "Rules + Memo + TED"
# - "Rules + TED"

## Example configuration
#################### syntax #################
## include:
##   - dataset: ["<dataset_name>"]
##     config: ["<configuration_name>"]
#
## Example:
## include:
##  - dataset: ["Carbon Bot"]
##    config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]
#
## Shortcut:
## You can use the "all" shortcut to include all available configurations or datasets
#
## Example: Use the "Sparse + EmbeddingIntent + ResponseSelector(bow)" configuration
## for all available datasets
## include:
##  - dataset: ["all"]
##    config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]
#
## Example: Use all available configurations for the "Carbon Bot" and "Sara" datasets
## and for the "Hermit" dataset use the "Sparse + DIET + ResponseSelector(T2T)" and
## "BERT + DIET + ResponseSelector(T2T)" configurations:
## include:
##  - dataset: ["Carbon Bot", "Sara"]
##    config: ["all"]
##  - dataset: ["Hermit"]
##    config: ["Sparse + DIET(seq) + ResponseSelector(t2t)", "BERT + DIET(seq) + ResponseSelector(t2t)"]
#
## Example: Define a branch name to check-out for a dataset repository. Default branch is 'main'
## dataset_branch: "test-branch"
## include:
##  - dataset: ["Carbon Bot", "Sara"]
##    config: ["all"]
##
## Shortcuts:
## You can use the "all" shortcut to include all available configurations or datasets.
## You can use the "all-nlu" shortcut to include all available NLU configurations or datasets.
## You can use the "all-core" shortcut to include all available core configurations or datasets.

include:
 - dataset: ["Carbon Bot"]
   config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]

```

github-actions · 2021-11-23T15:11:19Z

/modeltest

include:
 - dataset: ["Carbon Bot", "financial-demo", "Private 1"]
   config:
     - "BERT + DIET(seq) + ResponseSelector(t2t)"
     - "Spacy + DIET(seq) + ResponseSelector(t2t)"
     - "Sparse + BERT + DIET(seq) + ResponseSelector(t2t)"
     - "Sparse + DIET(seq) + ResponseSelector(t2t)"
     - "Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)"

github-actions · 2021-11-23T15:11:21Z

The model regression tests have started. It might take a while, please be patient.
As soon as results are ready you'll see a new comment with the results.

Used configuration can be found in the comment.

stale · 2022-04-16T06:54:52Z

This PR has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

CLAassistant · 2022-07-18T22:14:08Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ dakshvar22
❌ tczekajlo

tczekajlo seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

dakshvar22 added 12 commits July 12, 2021 17:50

bump rasa-sdk dep and min compat version

297f944

Merge branch '2.8.x' of github.com:RasaHQ/rasa into 2.8.x

ebc24f6

Merge branch '2.8.x' of github.com:RasaHQ/rasa into 2.8.x

e1218d4

Merge branch '2.8.x' of github.com:RasaHQ/rasa into 2.8.x

bf9b5e9

Merge branch '2.8.x' of github.com:RasaHQ/rasa into 2.8.x

5089c66

Merge branch '2.8.x' of github.com:RasaHQ/rasa into 2.8.x

e06441b

M erge branch '2.8.x' of github.com:RasaHQ/rasa into 2.8.x

895b38e

etry install --extras full

Merge branch '2.8.x' of github.com:RasaHQ/rasa into 2.8.x

bf6a969

Merge branch '2.8.x' of github.com:RasaHQ/rasa into 2.8.x

8dcbd7d

Merge branch '2.8.x' of github.com:RasaHQ/rasa into 2.8.x

0a87ef4

Merge branch '2.8.x' of github.com:RasaHQ/rasa into 2.8.x

81b62fc

remove grad filtering to see performance diffs

123d10c

dakshvar22 requested review from a team and jupyterjazz and removed request for a team and jupyterjazz October 13, 2021 19:01

dakshvar22 marked this pull request as draft October 13, 2021 19:02

dakshvar22 added runner:gpu status:model-regression-tests and removed status:model-regression-tests labels Oct 13, 2021

github-actions bot deleted a comment from dakshvar22 Oct 13, 2021

Remove livenessProbe for GPU runner

ea26e71

dakshvar22 added status:model-regression-tests and removed status:model-regression-tests labels Oct 22, 2021

github-actions bot removed status:model-regression-tests runner:gpu labels Oct 22, 2021

dakshvar22 added the status:model-regression-tests label Oct 22, 2021

github-actions bot removed the status:model-regression-tests label Oct 22, 2021

dakshvar22 mentioned this pull request Oct 22, 2021

Follow up on Tensorflow's recommendation for investigating the slowdown #9960

Closed

2 tasks

markus-hinsche added runner:gpu status:model-regression-tests labels Nov 17, 2021

github-actions bot removed status:model-regression-tests runner:gpu labels Nov 17, 2021

markus-hinsche added runner:gpu status:model-regression-tests labels Nov 17, 2021

github-actions bot removed status:model-regression-tests runner:gpu labels Nov 17, 2021

markus-hinsche mentioned this pull request Nov 18, 2021

Slowdown of tf.scan operation in TF 2.5 tensorflow/tensorflow#51393

Closed

markus-hinsche added the status:model-regression-tests label Nov 23, 2021

github-actions bot removed the status:model-regression-tests label Nov 23, 2021

stale bot added the stale label Apr 16, 2022

stale bot removed the stale label Jul 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove reg grad filtering to see performance diff #9867

Remove reg grad filtering to see performance diff #9867

dakshvar22 commented Oct 13, 2021 •

edited

github-actions bot commented Oct 18, 2021

github-actions bot commented Oct 22, 2021

github-actions bot commented Oct 22, 2021

github-actions bot commented Nov 17, 2021

github-actions bot commented Nov 17, 2021

github-actions bot commented Nov 23, 2021 •

edited

github-actions bot commented Nov 23, 2021

github-actions bot commented Nov 23, 2021

github-actions bot commented Nov 23, 2021

stale bot commented Apr 16, 2022

CLAassistant commented Jul 18, 2022 •

edited

Remove reg grad filtering to see performance diff #9867

Are you sure you want to change the base?

Remove reg grad filtering to see performance diff #9867

Conversation

dakshvar22 commented Oct 13, 2021 • edited

github-actions bot commented Oct 18, 2021

github-actions bot commented Oct 22, 2021

github-actions bot commented Oct 22, 2021

github-actions bot commented Nov 17, 2021

github-actions bot commented Nov 17, 2021

github-actions bot commented Nov 23, 2021 • edited

github-actions bot commented Nov 23, 2021

github-actions bot commented Nov 23, 2021

github-actions bot commented Nov 23, 2021

stale bot commented Apr 16, 2022

CLAassistant commented Jul 18, 2022 • edited

dakshvar22 commented Oct 13, 2021 •

edited

github-actions bot commented Nov 23, 2021 •

edited

CLAassistant commented Jul 18, 2022 •

edited