Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AugMemo DeLorean Inconsistency #10606

Closed
kedz opened this issue Dec 29, 2021 · 1 comment · Fixed by #10605
Closed

AugMemo DeLorean Inconsistency #10606

kedz opened this issue Dec 29, 2021 · 1 comment · Fixed by #10605
Assignees
Labels
area:rasa-oss 🎡 Anything related to the open source Rasa framework type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors.

Comments

@kedz
Copy link
Contributor

kedz commented Dec 29, 2021

Rasa Open Source version

2.8.7

Rasa SDK version

No response

Rasa X version

No response

Python version

3.8

What operating system are you using?

OSX

What happened?

A customer has a bot working as expected with Rasa 2.3.3 and Rasa 2.6.1. For those versions their stories follow the AugmentedMemoizationPolicy. While upgrading to Rasa 2.8.7 they noticed inconsistency within the behavior of the AugmentedMemoizationPolicy where for the same stories they were using previously the AugMemo policy was now losing track of the conversation and deferring to the TEDPolicy to make a prediction, which they observed was largely a low confidence prediction.

Potential Source of the Problem

A potential explanation for this is because of the change introduced in Rasa 2.8 where the DeLorean phase of AugMemo began to trim the tracker history according to the max_history prior to constructing the prediction states. Previously the entire tracker history was being used until this commit applying the tracker trimming.

When the AugmentedMemoizationPolicy is trained all the states contain a component detailing the last user intent, like this: {user: {"intent": "x"}, in addition to the various other action and slot events.

For the customer's case their story has a bunch of sequential actions occurring between user utterances. Once the number of sequential actions exceeds max_history when the DeLorean trims the tracker the last user utterance is lost. So, when the DeLorean constructs the prediction states from the trimmed tracker the last user intent is not included in those states. As a result AugMemo will not find a match because the states constructed during training and at prediction time do not contain the same information.

Reproducing the Error

The problem can be reproduced with the repo linked in this slack discussion, which contains a similar bot to what the customer is using. You can train a model and follow this conversation:

User: Hi
Bot: What can I help you with?
User: /technical_support

The bot will process actions up to action_set_customer_service_path where the bot will no longer be able to follow the AugmentedMemoizationPolicy to make predictions. Instead TED makes a low confidence prediction. The expectation is the bot would follow the checkpoint to predict action_check_customer_service_path. Reviewing the construction of the mcfly_tracker after action_set_customer_service_path the trimmed tracker no longer contains the last user utterance because there are now 6 actions occurring after the utterance and the utterance gets trimmed.

Command / Request

No response

Relevant log output

No response

@kedz kedz added type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors. area:rasa-oss 🎡 Anything related to the open source Rasa framework labels Dec 29, 2021
@kedz kedz self-assigned this Dec 29, 2021
@kedz
Copy link
Contributor Author

kedz commented Jan 3, 2022

@samsucik @JEM-Mosig reviewers

@kedz kedz closed this as completed Jan 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:rasa-oss 🎡 Anything related to the open source Rasa framework type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors.
Projects
None yet
4 participants