rasa data validate causes error with "action_two_stage_fallback"

[nyejon]

Rasa Open Source version

3.0.0

Rasa SDK version

No response

Rasa X version

No response

Python version

3.8

What operating system are you using?

OSX

What happened?

When using the two-stage fallback, I get the following error when running rasa data validate

UserWarning: The form 'action_two_stage_fallback' is used in the 'Activate two stage fallback' block, but it is not listed in the domain file. You should add it to your domain file!

I have this in my rules:

version: "3.0" rules:

• rule: Activate two stage fallback steps:
• intent: nlu_fallback
• action: action_two_stage_fallback
• active_loop: action_two_stage_fallback

Command / Request

rasa data validate

Relevant log output

No response

[sara-tagger]

Exalate commented:

sara-tagger commented:

Thanks for the issue, @pythenry will get back to you about it soon!

You may find help in the docs and the forum, too

🤗

[pythenry]

Exalate commented:

pythenry commented:

Hi @nyejon sincere apologies for the delayed response here. Can you post your domain file too please?

[Freemanlabs]

Exalate commented:

Freemanlabs commented:

Also experiencing the same problem here

[anthonydmg]

Exalate commented:

AnthonyMirandaGil commented:

I also had the same problem, is there a solution?

[anthonydmg]

Exalate commented:

AnthonyMirandaGil commented:

I have the same problem, when I add action_two_stage fallback in the rule.yml file and run the command rasa data validate I get the following error

UserWarning: The form 'action_two_stage_fallback' is used in the 'Implementation of the Two-Stage-Fallback' block, but it is not listed in the domain file. You should add it to your domain file! More info at https://rasa.com/docs/rasa/forms Project validation completed with errors.

I'm using the examples generated with rasa init and just adding the rule for the two stage fallback. These are my files:

domain.yml

version: "3.0"

intents:

• greet
• goodbye
• affirm
• deny
• mood_great
• mood_unhappy
• bot_challenge

responses:
utter_greet:

• text: "Hey! How are you?"

utter_cheer_up:

• text: "Here is something to cheer you up:"
  image: "https://i.imgur.com/nGF1K8f.jpg"

utter_did_that_help:

• text: "Did that help you?"

utter_happy:

• text: "Great, carry on!"

utter_goodbye:

• text: "Bye"

utter_iamabot:

• text: "I am a bot, powered by Rasa."

utter_default:

• text: I'm sorry, I can't help you.

utter_ask_rephrase:

• text: I'm sorry, I didn't quite understand that. Could you rephrase?

session_config:
session_expiration_time: 60
carry_over_slots_to_new_session: true

rule.ym

version: "3.0"
nlu:

• intent: greet
  examples: |

• hey

• hello

• hi

• hello there

• good morning

• good evening

• moin

• hey there

• let's go

• hey dude

• goodmorning

• goodevening

• good afternoon

• intent: goodbye
  examples: |

• cu

• good by

• cee you later

• good night

• bye

• goodbye

• have a nice day

• see you around

• bye bye

• see you later

• intent: affirm
  examples: |

• yes

• y

• indeed

• of course

• that sounds good

• correct

• intent: deny
  examples: |

• no

• n

• never

• I don't think so

• don't like that

• no way

• not really

• intent: mood_great
  examples: |

• perfect

• great

• amazing

• feeling like a king

• wonderful

• I am feeling very good

• I am great

• I am amazing

• I am going to save the world

• super stoked

• extremely good

• so so perfect

• so good

• so perfect

• intent: mood_unhappy
  examples: |

• my day was horrible

• I am sad

• I don't feel very well

• I am disappointed

• super sad

• I'm so sad

• sad

• very sad

• unhappy

• not good

• not very good

• extremly sad

• so saad

• so sad

• intent: bot_challenge
  examples: |

• are you a bot?

• are you a human?

• am I talking to a bot?

• am I talking to a human?

config.yml

recipe: default.v1

language: en

pipeline:

• name: WhitespaceTokenizer
• name: RegexFeaturizer
• name: LexicalSyntacticFeaturizer
• name: CountVectorsFeaturizer
• name: CountVectorsFeaturizer
  analyzer: char_wb
  min_ngram: 1
  max_ngram: 4
• name: DIETClassifier
  epochs: 100
• name: EntitySynonymMapper
• name: ResponseSelector
  epochs: 100
• name: FallbackClassifier
  threshold: 0.3
  ambiguity_threshold: 0.1

policies:

• name: MemoizationPolicy
• name: TEDPolicy
  max_history: 5
  epochs: 100
• name: RulePolicy

I am using version 3.0.5 of Rasa Open Source version

[pythenry]

Exalate commented:

pythenry commented:

Hi @AnthonyMirandaGil thank you for that example, apologies for the delay in response. I'll test this with your files and see if I can replicate it going forward.

[nyejon]

Exalate commented:

nyejon commented:

I also seem to get an error when the two stage fallback is actually run:

in _get_active_loop_ignored_intents
form_ignored_intents = domain.forms[active_loop_name].get(
KeyError: 'action_two_stage_fallback'

[TimoT18]

I also seem to get an error when the two stage fallback is actually run:

in _get_active_loop_ignored_intents
    form_ignored_intents = domain.forms[active_loop_name].get(
KeyError: 'action_two_stage_fallback'

I am running into the same error as described by nyejon when I reach the TwoStageFallbackAction and the user clickes one of the buttons. In my case the default settings for the TwoStageFallback were used as described here.

Rasa Version : 3.0.4
Minimum Compatible Version: 3.0.0
Rasa SDK Version : 3.0.2
Rasa X Version : None
Python Version : 3.7.11

[plc-dev]

@pythenry
Not sure when this change was made and if this is related, but the two stage fallback appears to be implemented as a form now and as such the _get_active_loop_ignored_intents method is called. This checks for ignored intents, which are usually defined in the domain when declaring a form.

I did not manage to find the part where the two stage fallback is registered as a form internally, but apparently it is not added to the mapping-dictionary that is checked in the previously mentioned method.

What fixed it for me was to (locally) expand the check in this line to the following:

if active_loop_name and active_loop_name != 'action_two_stage_fallback':

I guess the better way to fix this would be to add the entry to the checked dictionary with an empty list or whatever the desired behavior was meant to be.

[plattenschieber]

Hi all,

@akelad @wochinge maybe you have an idea, why rasa data validate fails when using The two_stage_fallback mechanism?

C:\DEV\miniconda\envs\rasa3\lib\site-packages\rasa\shared\utils\io.py:99: UserWarning: The form 'action_two_stage_fallback' is used in the 
'Implementation of the Two-Stage-Fallback' block, but it is not listed in the domain file. You should add it to your domain file!
  More info at https://rasa.com/docs/rasa/forms
Project validation completed with errors.

I tried it with both, Rasa Version 3.0.9/ Python 3.8.0 and 3.1.0/ Python 3.9.0. They both resulted in the same error.

We are very close to the documented example:

#rules.yml
- rule: Implementation of the Two-Stage-Fallback
  steps:
  - intent: nlu_fallback
  - action: action_two_stage_fallback
  - active_loop: action_two_stage_fallback
---
#config.yml
pipeline:
...
- name: FallbackClassifier
  threshold: 0.7
  ambiguity_threshold: 0.1
policies:
- name: RulePolicy
  core_fallback_threshold: 0.4
  core_fallback_action_name: action_default_fallback
  enable_fallback_prediction: true

[ktyborowski]

Running into the same issue. The two stage fallback also causes this error in normal rasa operation when using rasa run.

My version details:

Rasa Version      :         3.1.0
Minimum Compatible Version: 3.0.0
Rasa SDK Version  :         3.1.1
Rasa X Version    :         None
Python Version    :         3.8.9
Operating System  :         Windows-10-10.0.19041-SP0

[randomsven]

same problem for precisely same versions as pixpack report above while migrating a working bot from rasa 2.8.x - is there no resolution to this given the prevalence of the problem? Appears fallback implementation changed?

[ancalita]

Fix released in version 3.1.3 and 3.2.1

[cqx931]

still seeing KeyError: 'action_two_stage_fallback' when the user clicks one of the buttons. I'm also using the default settings for the TwoStageFallback were used as described here.

Rasa Version : 3.2.0
Python Version : 3.8.10

[ancalita]

@cqx931 That seems to be a different issue to the one described in the original issue. Please submit a separate bug report with more details please. Also note this bugfix was released in version 3.2.1: https://github.com/RasaHQ/rasa/releases/tag/3.2.1

[cqx931]

@ancalita thank you for pointing this bugfix. rasa data validate does work without problem now but the bot still hangs with the same key error in 3.2.1. Please see the ticket: #11294

[borisjota]

the bug has not been resolved in version 3.2.1 or 3.2.4, it seems to be a problem with the SLOTS, because when SLOTS are defined it is when the problem related to KeyError: 'action_two_stage_fallback' occurs

[ancalita]

@borisjota The bugfix released in version 3.2.1 targeted solely the rasa data validate reported issue here. I'm currently looking into the separate issue related to the KeyError: #11294

[plattenschieber]

@borisjota The bugfix released in version 3.2.1 targeted solely the rasa data validate reported issue here. I'm currently looking into the separate issue related to the KeyError.

Is there also an opened issue we could link here?