8705 response selector #9010

ka-bu · 2021-07-02T09:04:20Z

Note:

Continuing this on #9205. (closing this issue because we're merging this into 2.8.x via #9205)

Proposed changes:

closes ResponseSelector training fails with use_text_as_label and no transformer due to inconsistent data pre-processing #8705
- introduced some ignore_sequence* methods that control whether model_data and label_data creation methods should skip those features
- to cope with the missing features (empty) defaults are added later, so that sequence lengths can be computed and hence all expected inputs exist
some additional refactoring:
- there were two ways to tell _create_model_data that it's in "training" mode (passing a label_attribute or setting the trainng label) -> label_attribute is now an attribute and _create_label uses the training flag only
- there was a strange case in which, if in training mode but not doing intent classification, label tensors would have been created from an empty id list (_use_default_label_features)
- Tuple[tf.Tensor] --> Tuple[tf.Tensor,...] because this
- ...

Status (please check what you already did):

added some tests for the functionality
updated the documentation
updated the changelog (please check changelog for instructions)
reformat files using black (please check Readme for instructions)

…add test

JEM-Mosig

Well done! 💯
I have a few minor comments. Before we can merge this, please also

add a change log entry
run NLU model regression tests with all configurations on all datasets

rasa/nlu/classifiers/diet_classifier.py

rasa/utils/tensorflow/models.py

rasa/utils/tensorflow/rasa_layers.py

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

JEM-Mosig · 2021-07-16T17:55:52Z

/modeltest

# Run "Rules + TED" config on all Core datasets.
dataset_branch: "variable_random_seed"
include:
  - dataset: ["all"]
    config: ["all-nlu"]

JEM-Mosig

Getting there... :)
Some conflicts with main need to be resolved.

rasa/nlu/classifiers/diet_classifier.py

JEM-Mosig · 2021-07-16T19:49:54Z

rasa/nlu/classifiers/diet_classifier.py


-        if training and label_attribute is not None:
+        if training:


Suggested change

if training:

_create_model_data is also used during predict

JEM-Mosig · 2021-07-16T20:24:44Z

rasa/utils/tensorflow/model_data.py

+            `FeatureArray`s in case there is no subkey), or
+            an empty list in case the requested data cannot be found
+        """
+        default = []


You mean most get methods have None as a default and we'd have to write default = [] which would be unusual? And if we wrote default = None then we'd have to modify all calls.

rasa/utils/tensorflow/model_data_utils.py

rasa/utils/tensorflow/rasa_layers.py

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

…sserts,

github-actions · 2021-07-19T14:35:59Z

Hey @ka-bu! 👋 To run model regression tests, comment with the /modeltest command and a configuration.

Tips 💡: The model regression test will be run on push events. You can re-run the tests by re-add status:model-regression-tests label or use a Re-run jobs button in Github Actions workflow.

Tips 💡: Every time when you want to change a configuration you should edit the comment with the previous configuration.

You can copy this in your comment and customize:

/modeltest

```yml
##########
## Available datasets
##########
# - "Carbon Bot" (NLU)
# - "Hermit" (NLU)
# - "Private 1" (NLU)
# - "Private 2" (NLU)
# - "Private 3" (NLU)
# - "Sara" (NLU, Core)
# - "financial-demo" (NLU, Core)
# - "helpdesk-assistant" (NLU, Core)

##########
## Available NLU configurations
##########
# - "BERT + DIET(bow) + ResponseSelector(bow)"
# - "BERT + DIET(seq) + ResponseSelector(t2t)"
# - "Spacy + DIET(bow) + ResponseSelector(bow)"
# - "Spacy + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + BERT + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + BERT + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + DIET(seq) + ResponseSelector(t2t)"
# - "Sparse + Spacy + DIET(bow) + ResponseSelector(bow)"
# - "Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)"

##########
## Available Core configurations
##########
# - "Rules"
# - "Rules + AugMemo"
# - "Rules + AugMemo + TED"
# - "Rules + Memo"
# - "Rules + Memo + TED"
# - "Rules + TED"

## Example configuration
#################### syntax #################
## include:
##   - dataset: ["<dataset_name>"]
##     config: ["<configuration_name>"]
#
## Example:
## include:
##  - dataset: ["Carbon Bot"]
##    config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]
#
## Shortcut:
## You can use the "all" shortcut to include all available configurations or datasets
#
## Example: Use the "Sparse + EmbeddingIntent + ResponseSelector(bow)" configuration
## for all available datasets
## include:
##  - dataset: ["all"]
##    config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]
#
## Example: Use all available configurations for the "Carbon Bot" and "Sara" datasets
## and for the "Hermit" dataset use the "Sparse + DIET + ResponseSelector(T2T)" and
## "BERT + DIET + ResponseSelector(T2T)" configurations:
## include:
##  - dataset: ["Carbon Bot", "Sara"]
##    config: ["all"]
##  - dataset: ["Hermit"]
##    config: ["Sparse + DIET(seq) + ResponseSelector(t2t)", "BERT + DIET(seq) + ResponseSelector(t2t)"]
#
## Example: Define a branch name to check-out for a dataset repository. Default branch is 'main'
## dataset_branch: "test-branch"
## include:
##  - dataset: ["Carbon Bot", "Sara"]
##    config: ["all"]
##
## Shortcuts:
## You can use the "all" shortcut to include all available configurations or datasets.
## You can use the "all-nlu" shortcut to include all available NLU configurations or datasets.
## You can use the "all-core" shortcut to include all available core configurations or datasets.

include:
 - dataset: ["Carbon Bot"]
   config: ["Sparse + DIET(bow) + ResponseSelector(bow)"]

```

github-actions · 2021-07-19T14:36:02Z

/modeltest

include:
  - dataset: ["all"]
    config: ["all-nlu"]

github-actions · 2021-07-19T14:36:04Z

The model regression tests have started. It might take a while, please be patient.
As soon as results are ready you'll see a new comment with the results.

Used configuration can be found in the comment.

github-actions · 2021-07-19T21:46:41Z

Commit: 0502ae9, The full report is available as an artifact.

Dataset: Carbon Bot, Dataset repository branch: main, commit: de8d01c055753aa800f188f1dad8dc3e752bd7cb

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `1m27s`, train: `4m37s`, total: `6m3s`	0.7942 (0.00)	0.7529 (0.00)	0.5382 (0.00)
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `1m46s`, train: `4m4s`, total: `5m50s`	0.8000 (0.00)	0.7766 (0.00)	0.5430 (0.00)
`Sparse + BERT + DIET(bow) + ResponseSelector(bow)` test: `1m33s`, train: `4m19s`, total: `5m52s`	0.7845 (-0.00)	0.7529 (0.00)	0.5563 (-0.03)
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `1m54s`, train: `4m42s`, total: `6m36s`	0.7806 (0.00)	0.7935 (0.00)	0.5629 (0.02)
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `41s`, train: `2m33s`, total: `3m13s`	0.7417 (-0.01)	0.7529 (0.00)	0.5183 (0.01)
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `1m2s`, train: `3m56s`, total: `4m57s`	0.7398 (0.00)	0.6819 (0.00)	0.5298 (0.01)

Dataset: Hermit, Dataset repository branch: main, commit: dac7e1dd73194d12861829c7175ee9615e92ff6e

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `2m47s`, train: `19m56s`, total: `22m42s`	0.8978 (0.00)	0.7504 (0.00)	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `3m6s`, train: `12m30s`, total: `15m35s`	0.8941 (0.00)	0.8060 (0.00)	`no data`
`Sparse + BERT + DIET(bow) + ResponseSelector(bow)` test: `2m48s`, train: `22m31s`, total: `25m18s`	0.8690 (-0.00)	0.7504 (0.00)	`no data`
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `3m6s`, train: `13m19s`, total: `16m25s`	0.8699 (-0.00)	0.7985 (0.00)	`no data`
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `1m8s`, train: `19m27s`, total: `20m34s`	0.8327 (0.02)	0.7504 (0.00)	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `1m25s`, train: `12m14s`, total: `13m39s`	0.8364 (0.00)	0.7578 (-0.00)	`no data`

Dataset: Private 1, Dataset repository branch: main, commit: dac7e1dd73194d12861829c7175ee9615e92ff6e

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `1m56s`, train: `3m23s`, total: `5m19s`	0.9096 (0.00)	0.9612 (0.00)	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m20s`, train: `3m12s`, total: `5m32s`	0.9137 (0.00)	0.9709 (0.00)	`no data`
`Spacy + DIET(bow) + ResponseSelector(bow)` test: `35s`, train: `2m42s`, total: `3m16s`	0.8420 (0.00)	0.9574 (0.00)	`no data`
`Spacy + DIET(seq) + ResponseSelector(t2t)` test: `57s`, train: `3m13s`, total: `4m9s`	0.8555 (0.00)	0.9402 (0.00)	`no data`
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `28s`, train: `2m58s`, total: `3m26s`	0.8971 (0.00)	0.9612 (0.00)	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `51s`, train: `3m4s`, total: `3m54s`	0.9012 (0.00)	0.9718 (0.00)	`no data`
`Sparse + Spacy + DIET(bow) + ResponseSelector(bow)` test: `38s`, train: `3m37s`, total: `4m14s`	0.8971 (0.00)	0.9574 (0.00)	`no data`
`Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)` test: `59s`, train: `3m28s`, total: `4m26s`	0.9033 (0.01)	0.9727 (0.00)	`no data`

Dataset: Private 2, Dataset repository branch: main, commit: dac7e1dd73194d12861829c7175ee9615e92ff6e

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `1m58s`, train: `10m16s`, total: `12m13s`	0.8745 (0.00)	`no data`	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `2m5s`, train: `5m16s`, total: `7m21s`	0.8830 (0.00)	`no data`	`no data`
`Spacy + DIET(bow) + ResponseSelector(bow)` test: `39s`, train: `5m13s`, total: `5m52s`	0.7253 (0.00)	`no data`	`no data`
`Spacy + DIET(seq) + ResponseSelector(t2t)` test: `47s`, train: `5m7s`, total: `5m54s`	0.7811 (0.00)	`no data`	`no data`
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `35s`, train: `4m24s`, total: `4m58s`	0.8466 (-0.01)	`no data`	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `41s`, train: `4m26s`, total: `5m6s`	0.8530 (0.00)	`no data`	`no data`
`Sparse + Spacy + DIET(bow) + ResponseSelector(bow)` test: `44s`, train: `6m38s`, total: `7m22s`	0.8509 (-0.02)	`no data`	`no data`
`Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)` test: `52s`, train: `5m36s`, total: `6m28s`	0.8498 (0.00)	`no data`	`no data`

Dataset: Private 3, Dataset repository branch: main, commit: dac7e1dd73194d12861829c7175ee9615e92ff6e

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `1m4s`, train: `1m0s`, total: `2m4s`	0.9177 (0.00)	`no data`	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `1m4s`, train: `45s`, total: `1m49s`	0.8436 (0.00)	`no data`	`no data`
`Spacy + DIET(bow) + ResponseSelector(bow)` test: `36s`, train: `50s`, total: `1m27s`	0.6132 (0.00)	`no data`	`no data`
`Spacy + DIET(seq) + ResponseSelector(t2t)` test: `42s`, train: `40s`, total: `1m21s`	0.6214 (0.00)	`no data`	`no data`
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `33s`, train: `57s`, total: `1m29s`	0.8683 (0.00)	`no data`	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `38s`, train: `39s`, total: `1m17s`	0.8642 (0.00)	`no data`	`no data`
`Sparse + Spacy + DIET(bow) + ResponseSelector(bow)` test: `37s`, train: `1m8s`, total: `1m44s`	0.8436 (0.00)	`no data`	`no data`
`Sparse + Spacy + DIET(seq) + ResponseSelector(t2t)` test: `42s`, train: `47s`, total: `1m29s`	0.8601 (0.00)	`no data`	`no data`

Dataset: Sara, Dataset repository branch: main, commit: dac7e1dd73194d12861829c7175ee9615e92ff6e

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `4m38s`, train: `5m27s`, total: `10m5s`	0.7135 (-0.14)	0.7895 (-0.08)	0.7898 (-0.10)
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `5m5s`, train: `4m20s`, total: `9m25s`	0.7087 (-0.14)	0.7903 (-0.08)	0.7767 (-0.09)
`Sparse + BERT + DIET(bow) + ResponseSelector(bow)` test: `4m55s`, train: `7m43s`, total: `12m38s`	0.6875 (-0.18)	0.7895 (-0.08)	0.8093 (-0.07)
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `5m29s`, train: `5m31s`, total: `11m0s`	0.6943 (-0.18)	0.7820 (-0.11)	0.8031 (-0.10)
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `1m36s`, train: `5m38s`, total: `7m13s`	0.6659 (-0.15)	0.7895 (-0.08)	0.7867 (-0.08)
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `2m4s`, train: `4m28s`, total: `6m31s`	0.6740 (-0.17)	0.7723 (-0.05)	0.7938 (-0.06)

Dataset: financial-demo, Dataset repository branch: main (external repository), commit: 27cc9e1727f800364de6feaf8c5a836cebd822ff
Configuration repository branch: main

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `27s`, train: `47s`, total: `1m14s`	1.0000 (0.00)	0.8333 (0.00)	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `37s`, train: `53s`, total: `1m29s`	1.0000 (0.00)	0.8333 (0.00)	`no data`
`Sparse + BERT + DIET(bow) + ResponseSelector(bow)` test: `28s`, train: `58s`, total: `1m25s`	1.0000 (0.00)	0.8333 (0.00)	`no data`
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `37s`, train: `57s`, total: `1m33s`	1.0000 (0.00)	0.8800 (0.00)	`no data`
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `20s`, train: `41s`, total: `1m1s`	0.9643 (0.00)	0.8333 (0.00)	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `30s`, train: `48s`, total: `1m17s`	0.9643 (0.00)	0.8800 (0.00)	`no data`

Dataset: helpdesk-assistant, Dataset repository branch: main (external repository), commit: 7541291c31df0219ad343808c963b4642a8332cb
Configuration repository branch: main

Configuration	Intent Classification Micro F1	Entity Recognition Micro F1	Response Selection Micro F1
`BERT + DIET(bow) + ResponseSelector(bow)` test: `22s`, train: `42s`, total: `1m4s`	1.0000 (0.00)	`no data`	`no data`
`BERT + DIET(seq) + ResponseSelector(t2t)` test: `32s`, train: `45s`, total: `1m16s`	1.0000 (0.00)	`no data`	`no data`
`Sparse + BERT + DIET(bow) + ResponseSelector(bow)` test: `22s`, train: `49s`, total: `1m11s`	1.0000 (0.00)	`no data`	`no data`
`Sparse + BERT + DIET(seq) + ResponseSelector(t2t)` test: `32s`, train: `48s`, total: `1m19s`	1.0000 (0.00)	`no data`	`no data`
`Sparse + DIET(bow) + ResponseSelector(bow)` test: `16s`, train: `36s`, total: `51s`	1.0000 (0.00)	`no data`	`no data`
`Sparse + DIET(seq) + ResponseSelector(t2t)` test: `26s`, train: `41s`, total: `1m6s`	1.0000 (0.00)	`no data`	`no data`

dakshvar22 · 2021-07-21T10:03:04Z

Should all the configurations with *t2t.yml suffix use use_text_as_label: True for ResponseSelector and then model regression tests should be run?
Eventually, we would like to use that parameter in all t2t configs as a regular way to test if this breaks going forward.

ka-bu · 2021-07-28T12:08:21Z

Closing because we're merging this into 2.8.x via #9205

ka-bu added 3 commits July 1, 2021 21:42

add fix and some refactoring

10bde59

merge main

4899f57

change (label_id/default_label handling)

e83d942

ka-bu requested a review from JEM-Mosig July 2, 2021 09:04

ka-bu added runner:gpu status:model-regression-tests labels Jul 2, 2021

ka-bu added 2 commits July 2, 2021 11:22

minor (linebreaks)

8cbbe8e

minor (lint-docstrings)

fe3119d

ka-bu removed runner:gpu status:model-regression-tests labels Jul 2, 2021

ka-bu added 2 commits July 2, 2021 12:11

Merge branch 'main' into 8705-response-selector

c92776b

fix (sparse feature sizes vs filter in featurize_training_examples); …

e6c7fc2

…add test

RasaHQ deleted a comment from github-actions bot Jul 2, 2021

ka-bu added 2 commits July 2, 2021 14:12

fix (rm debug)

909d493

fix/change test (rasa_layers)

8f8e348

ka-bu marked this pull request as ready for review July 2, 2021 13:11

ka-bu requested a review from a team July 2, 2021 13:11

JEM-Mosig suggested changes Jul 9, 2021

View reviewed changes

samsucik mentioned this pull request Jul 12, 2021

ResponseSelector training fails with use_text_as_label and no transformer due to inconsistent data pre-processing #8705

Closed

2 tasks

ka-bu and others added 11 commits July 12, 2021 10:39

Update rasa/nlu/classifiers/diet_classifier.py

e0c0d5c

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

Update rasa/nlu/classifiers/diet_classifier.py

8eb0764

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

Update rasa/nlu/selectors/response_selector.py

9cabbc4

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

Update rasa/nlu/selectors/response_selector.py

fb441fb

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

Update rasa/nlu/selectors/response_selector.py

63ddae5

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

Update rasa/utils/tensorflow/model_data.py

e8b7ace

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

Update rasa/utils/tensorflow/model_data.py

9d66633

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

Update rasa/utils/tensorflow/models.py

2c8c03f

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

Update rasa/utils/tensorflow/models.py

555e18f

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

Update rasa/utils/tensorflow/rasa_layers.py

869794c

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

Update rasa/nlu/classifiers/diet_classifier.py

61641a2

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

fix (add missing type annotation)

de46204

ka-bu added runner:gpu status:model-regression-tests labels Jul 14, 2021

RasaHQ deleted a comment from github-actions bot Jul 15, 2021

JEM-Mosig removed runner:gpu status:model-regression-tests labels Jul 16, 2021

JEM-Mosig added runner:gpu status:model-regression-tests labels Jul 16, 2021

JEM-Mosig suggested changes Jul 16, 2021

View reviewed changes

ka-bu and others added 9 commits July 19, 2021 09:23

Update rasa/nlu/classifiers/diet_classifier.py

5e69baf

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

Update rasa/nlu/classifiers/diet_classifier.py

846c0da

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

Update rasa/nlu/classifiers/diet_classifier.py

2492e76

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

Update rasa/utils/tensorflow/rasa_layers.py

de8c8b5

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

Update rasa/nlu/classifiers/diet_classifier.py

87c9686

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

Update rasa/nlu/classifiers/diet_classifier.py

876f82b

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

Update rasa/nlu/classifiers/diet_classifier.py

abeca8b

Co-authored-by: Johannes E. M. Mosig <j.mosig@rasa.com>

add/change (test for add_lengths, rearranged _check_* functions, rm a…

255bb27

…sserts,

merge main

7441256

github-actions bot removed status:model-regression-tests runner:gpu labels Jul 19, 2021

ka-bu requested a review from JEM-Mosig July 20, 2021 07:33

ka-bu mentioned this pull request Jul 26, 2021

8705 response selector (2.8.x) #9205

Closed

4 tasks

ka-bu closed this Jul 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8705 response selector #9010

8705 response selector #9010

ka-bu commented Jul 2, 2021 •

edited

JEM-Mosig left a comment

JEM-Mosig commented Jul 16, 2021

JEM-Mosig left a comment •

edited

JEM-Mosig Jul 16, 2021

ka-bu Jul 19, 2021

JEM-Mosig Jul 16, 2021

github-actions bot commented Jul 19, 2021

github-actions bot commented Jul 19, 2021

github-actions bot commented Jul 19, 2021

github-actions bot commented Jul 19, 2021

dakshvar22 commented Jul 21, 2021

ka-bu commented Jul 28, 2021

8705 response selector #9010

8705 response selector #9010

Conversation

ka-bu commented Jul 2, 2021 • edited

JEM-Mosig left a comment

Choose a reason for hiding this comment

JEM-Mosig commented Jul 16, 2021

JEM-Mosig left a comment • edited

Choose a reason for hiding this comment

JEM-Mosig Jul 16, 2021

Choose a reason for hiding this comment

ka-bu Jul 19, 2021

Choose a reason for hiding this comment

JEM-Mosig Jul 16, 2021

Choose a reason for hiding this comment

github-actions bot commented Jul 19, 2021

github-actions bot commented Jul 19, 2021

github-actions bot commented Jul 19, 2021

github-actions bot commented Jul 19, 2021

dakshvar22 commented Jul 21, 2021

ka-bu commented Jul 28, 2021

ka-bu commented Jul 2, 2021 •

edited

JEM-Mosig left a comment •

edited