Reduce the constrain_similarities warning in the CI Tests

[Imod7]

Python version: Python 3.6 Python 3.7 Python 3.8

Operating system (windows, osx, ...): Ubuntu Windows

Issue: This Issue is part of the effort to close the Issue#7738 hence to reduce the amount of warnings that are raised when running all tests in the CI. From a first investigation, it seems that this warning ( UserWarning: constrain_similarities is set to False. It is recommended to set it to True... ) for which the current Issue was opened, had 160 occurrences but in a latest CI run it appears in only 76 tests.

Warning (including full traceback):

2021-03-09T05:38:55.1847117Z rasa/utils/train_utils.py:451: 76 tests with warnings 2021-03-09T05:38:55.1848852Z /home/runner/work/rasa/rasa/rasa/utils/train_utils.py:451: UserWarning: constrain_similarities is set to False. It is recommended to set it to True when using cross-entropy loss. It will be set to True by default, Rasa Open Source 3.0.0 onwards.

Possible Solutions:

• The default value for the constant variable CONSTRAIN_SIMILARITIES is set to False so maybe setting it to True eliminates the warning.
• However, it needs to be checked what might be affected in other parts of the code or in general functionality from applying this change.

Command or request that led to error:

• The warning can be found in any of the following CI tests log files :
  • 1_Run Tests (ubuntu-latest, 3.6, test-non-training).txt
  • 3_Run Tests (ubuntu-latest, 3.7, test-non-training).txt
  • 5_Run Tests (ubuntu-latest, 3.8, test-non-training).txt
  • 10_Run Tests (windows-latest, 3.7, test-training).txt
• Also, the warning can be reproduced when running one of the tests below:

pytest tests/core/policies/test_ted_policy.py::TestTEDPolicyMargin::test_normalization pytest tests/core/policies/test_ted_policy.py::TestTEDPolicyMargin::test_prediction_on_empty_tracker pytest tests/nlu/classifiers/test_diet_classifier.py::test_check_labels_features_exist pytest tests/nlu/test_evaluation.py::test_response_selector_present

Definition of Done:

• The constrain_similarities warning appears in only the cases where it is explicitly tested for False.
• In all the cases that the constrain_similarities takes a default value then this should be the recommended one.

[iurisevero]

Exalate commented:

iurisevero commented:

Hey there, I worked a little on this issue and it looks like changing the default value of constrain_similarities to true at TEDPolicy, DietClassifier, and Response Selector fixed it.

All UserWarning: constrain_similarities is set to False. It is recommended to set it to True... warnings have gone and the CI passed, so I don't think it broke anything.

[rgstephens]

Exalate commented:

rgstephens commented:

Under 2.6.1 this warning is being thrown even when constrain_similarities isn't set.

Does this mean the default is set to false?

[Imod7]

Exalate commented:

Imod7 commented:

Yes! The default is set to False and for the time being this cannot be changed until Rasa 3.0 because as the research team mentioned this would require also changes in the default loss function.