End-to-End (e2e) Evaluation crashes if entity is not extracted

[enryH]

Rasa Core version: 0.12.2

Python version: 3.6.5

Operating system (windows, osx, ...): Windows

Issue: End-to-End (e2e) Evaluation crashes if entity is not extracted.

Content of domain file (if used & relevant):

slots:
  location:
    type: text
  object:
    type: text

intents:
 - greet
 - goodbye
 - inform
 - question
 - mood_unhappy
 - deny
 - confirm
 - other

entities:
 - location
 - object

...

I have a problem using end-2-end (e2e) evaluations. If I specify a location - here "work" - which was not in the training data (and therefore is most likely not recognized), the procedure breaks down.

Example working and not:

## story is working
* greet: Hoi
  - utter_greet
* inform: At [home](location)

## story is NOT working
* greet: Hoi
  - utter_greet
* inform: At [work](location)

###Traceback

(rasastack) C:\Users\enryh\Desktop\Chatbot_Textclassification\rasa_chatbot>python -m rasa_core.evaluate default --core models/dialogue --nlu models/nlu/default/ --stories data/testdata_stories_e2e.md -o test/ --e2e
C:\Users\enryh\Envs\rasastack\lib\importlib\_bootstrap.py:219: RuntimeWarning: cymem.cymem.Pool size changed, may indicate binary incompatibility. Expected 48 from C header, got 64 from PyObject
  return f(*args, **kwds)
C:\Users\enryh\Envs\rasastack\lib\importlib\_bootstrap.py:219: RuntimeWarning: cymem.cymem.Address size changed, may indicate binary incompatibility. Expected 24 from C header, got 40 from PyObject
  return f(*args, **kwds)
INFO:rasa_nlu.components:Added 'nlp_spacy' to component cache. Key 'nlp_spacy-de'.
C:\Users\enryh\Envs\rasastack\lib\site-packages\pykwalify\core.py:99: UnsafeLoaderWarning:
The default 'Loader' for 'load(stream)' without further arguments can be unsafe.
Use 'load(stream, Loader=ruamel.yaml.Loader)' explicitly if that is OK.
Alternatively include the following in your code:

  import warnings
  warnings.simplefilter('ignore', ruamel.yaml.error.UnsafeLoaderWarning)

In most other cases you should consider using 'safe_load(stream)'
  data = yaml.load(stream)
2018-11-26 16:09:56.536190: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
Processed Story Blocks: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 491.42it/s, # trackers=1]
INFO:__main__:Evaluating 1 stories
Progress:
  0%|                                                                                                                                        | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "c:\users\enryh\appdata\local\programs\python\python36\Lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\enryh\appdata\local\programs\python\python36\Lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\enryh\Envs\rasastack\lib\site-packages\rasa_core\evaluate.py", line 666, in <module>
    cmdline_arguments.e2e)
  File "C:\Users\enryh\Envs\rasastack\lib\site-packages\rasa_core\evaluate.py", line 491, in run_story_evaluation
    use_e2e)
  File "C:\Users\enryh\Envs\rasastack\lib\site-packages\rasa_core\evaluate.py", line 433, in collect_story_predictions
    fail_on_prediction_errors, use_e2e)
  File "C:\Users\enryh\Envs\rasastack\lib\site-packages\rasa_core\evaluate.py", line 378, in _predict_tracker_actions
    fail_on_prediction_errors
  File "C:\Users\enryh\Envs\rasastack\lib\site-packages\rasa_core\evaluate.py", line 337, in _collect_action_executed_predictions
    action, policy, confidence = processor.predict_next_action(partial_tracker)
  File "C:\Users\enryh\Envs\rasastack\lib\site-packages\rasa_core\processor.py", line 168, in predict_next_action
    probabilities, policy = self._get_next_action_probabilities(tracker)
  File "C:\Users\enryh\Envs\rasastack\lib\site-packages\rasa_core\processor.py", line 478, in _get_next_action_probabilities
    tracker, self.domain)
  File "C:\Users\enryh\Envs\rasastack\lib\site-packages\rasa_core\policies\ensemble.py", line 281, in probabilities_using_best_policy
    probabilities = p.predict_action_probabilities(tracker, domain)
  File "C:\Users\enryh\Envs\rasastack\lib\site-packages\rasa_core\policies\memoization.py", line 194, in predict_action_probabilities
    [tracker], domain)
  File "C:\Users\enryh\Envs\rasastack\lib\site-packages\rasa_core\featurizers.py", line 648, in prediction_states
    for tracker in trackers]
  File "C:\Users\enryh\Envs\rasastack\lib\site-packages\rasa_core\featurizers.py", line 648, in <listcomp>
    for tracker in trackers]
  File "C:\Users\enryh\Envs\rasastack\lib\site-packages\rasa_core\featurizers.py", line 291, in _create_states
    states = tracker.past_states(domain)
  File "C:\Users\enryh\Envs\rasastack\lib\site-packages\rasa_core\trackers.py", line 150, in past_states
    generated_states = domain.states_for_tracker_history(self)
  File "C:\Users\enryh\Envs\rasastack\lib\site-packages\rasa_core\domain.py", line 475, in states_for_tracker_history
    tracker.generate_all_prior_trackers()]
  File "C:\Users\enryh\Envs\rasastack\lib\site-packages\rasa_core\domain.py", line 474, in <listcomp>
    return [self.get_active_states(tr) for tr in
  File "C:\Users\enryh\Envs\rasastack\lib\site-packages\rasa_core\domain.py", line 466, in get_active_states
    state_dict = self.get_parsing_states(tracker)
  File "C:\Users\enryh\Envs\rasastack\lib\site-packages\rasa_core\domain.py", line 406, in get_parsing_states
    key = "entity_{0}".format(entity["entity"])
KeyError: 'entity'

Can anyone reproduce this error message?

Best regards,
Henry

[akelad]

Thanks for raising this issue, @ricwo will get back to you about it soon.

[ricwo]

@enryH I can confirm that the KeyError is thrown on unseen entities. This is a bug and we'll look into fixing that soon! Thanks for pointing this out

[enryH]

@ricwo: Thanks for looking into it. The Traceback suggests that the problem might be tedious to solve, so I did not dare to follow it along and try to debug it...