Merge branch 'master' into safe_load_spacy

CHANGELOG.rst

@@ -6,21 +6,32 @@ Rasa Change Log
 All notable changes to this project will be documented in this file.
 This project adheres to `Semantic Versioning`_ starting with version 1.0.
 
-[Unreleased 1.0.3.aX] - `master`_
+[Unreleased 1.0.4.aX] - `master`_
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Added
 -----
 
 Changed
 -------
+- NLU evaluation in cross-validation mode now also provides intent/entity reports,
+  confusion matrix, etc.
 
 Removed
 -------
 
 Fixed
 -----
 
+[1.0.3] - 2019-05-30
+^^^^^^^^^^^^^^^^^^^^
+
+Fixed
+-----
+- non-ascii characters render correctly in stories generated from interactive learning
+- validate domain file before usage, e.g. print proper error messages if domain file
+  is invalid instead of raising errors
+
 [1.0.2] - 2019-05-29
 ^^^^^^^^^^^^^^^^^^^^
 
@@ -32,13 +43,12 @@ Added
 
 Fixed
 -----
-- fix lookup table files failed to load issues/3622 
+- fix lookup table files failed to load issues/3622
 - buttons can now be properly selected during cmdline chat or when in interactive learning
 - set slots correctly when events are added through the API
 - mapping policy no longer ignores NLU threshold
 - mapping policy priority is correctly persisted
-- validate domain file before usage, e.g. print proper error messages if domain file is invalid instead of raising
-  errors
+
 
 [1.0.1] - 2019-05-21
 ^^^^^^^^^^^^^^^^^^^^

docs/core/actions.rst

@@ -11,7 +11,7 @@ Actions
 Actions are the things your bot runs in response to user input.
 There are three kinds of actions in Rasa Core:
 
- 1. **Default actions**: ``action_listen``, ``action_back``, ``action_restart``,
+ 1. **Default actions**: e.g. ``action_listen``, ``action_restart``,
     ``action_default_fallback``
  2. **Utterance actions**: start with ``utter_``, just send a message
     to the user
@@ -137,28 +137,51 @@ events in :ref:`Events <events>`.
 Default Actions
 ---------------
 
-There are three default actions:
-
-+-----------------------------+------------------------------------------------+
-| ``action_listen``           | stop predicting more actions and wait for user |
-|                             | input                                          |
-+-----------------------------+------------------------------------------------+
-| ``action_restart``          | reset the whole conversation, usually triggered|
-|                             | by using ``/restart``                          |
-+-----------------------------+------------------------------------------------+
-| ``action_default_fallback`` | undoes the last user message (as if the user   |
-|                             | did not send it) and utters a message that the |
-|                             | bot did not understand. See                    |
-|                             | :ref:`fallback-actions`.                       |
-+-----------------------------+------------------------------------------------+
+There are eight default actions:
+
++-----------------------------------+------------------------------------------------+
+| ``action_listen``                 | Stop predicting more actions and wait for user |
+|                                   | input.                                         |
++-----------------------------------+------------------------------------------------+
+| ``action_restart``                | Reset the whole conversation. Can be triggered |
+|                                   | during a conversation by entering ``/restart`` |
+|                                   | if the :ref:`mapping-policy` is included in    |
+|                                   | the policy configuration.                      |
++-----------------------------------+------------------------------------------------+
+| ``action_default_fallback``       | Undo the last user message (as if the user did |
+|                                   | not send it and the bot did not react) and     |
+|                                   | utter a message that the bot did not           |
+|                                   | understand. See :ref:`fallback-actions`.       |
++-----------------------------------+------------------------------------------------+
+| ``action_deactivate_form``        | Deactivate the active form and reset the       |
+|                                   | requested slot.                                |
+|                                   | See also :ref:`section_unhappy`.               |
++-----------------------------------+------------------------------------------------+
+| ``action_revert_fallback_events`` | Revert events that occurred during the         |
+|                                   | TwoStageFallbackPolicy.                        |
+|                                   | See :ref:`fallback-actions`.                   |
++-----------------------------------+------------------------------------------------+
+| ``action_default_ask_affirmation``| Ask the user to affirm their intent.           |
+|                                   | It is suggested to overwrite this default      |
+|                                   | action with a custom action to have more       |
+|                                   | meaningful prompts.                            |
++-----------------------------------+------------------------------------------------+
+| ``action_default_ask_rephrase``   | Ask the user to rephrase their intent.         |
++-----------------------------------+------------------------------------------------+
+| ``action_back``                   | Undo the last user message (as if the user did |
+|                                   | not send it and the bot did not react).        |
+|                                   | Can be triggered during a conversation by      |
+|                                   | entering ``/back`` if the MappingPolicy is     |
+|                                   | included in the policy configuration.          |
++-----------------------------------+------------------------------------------------+
 
 All the default actions can be overwritten. To do so, add the action name
 to the list of actions in your domain:
 
 .. code-block:: yaml
 
   actions:
-  - action_listen
+  - action_default_ask_affirmation
 
 Rasa Core will then call your action endpoint and treat it as every other
 custom action.

docs/core/policies.rst

@@ -342,6 +342,7 @@ training data. It predicts the next action with confidence ``1.0``
 if this exact conversation exists in the training data, otherwise it
 predicts ``None`` with confidence ``0.0``.
 
+.. _mapping-policy:
 
 Mapping Policy
 ^^^^^^^^^^^^^^

rasa/cli/test.py

@@ -99,7 +99,7 @@ def test_nlu(args: argparse.Namespace) -> None:
         print ("No model specified. Model will be trained using cross validation.")
         config = get_validated_path(args.config, "config", DEFAULT_CONFIG_PATH)
 
-        test_nlu_with_cross_validation(config, nlu_data, args.folds)
+        test_nlu_with_cross_validation(config, nlu_data, vars(args))
 
 
 def test(args: argparse.Namespace):

rasa/core/events/__init__.py

@@ -276,7 +276,8 @@ def as_story_string(self, e2e=False):
         if self.intent:
             if self.entities:
                 ent_string = json.dumps(
-                    {ent["entity"]: ent["value"] for ent in self.entities}
+                    {ent["entity"]: ent["value"] for ent in self.entities},
+                    ensure_ascii=False,
                 )
             else:
                 ent_string = ""
@@ -425,7 +426,7 @@ def __eq__(self, other):
             return (self.key, self.value) == (other.key, other.value)
 
     def as_story_string(self):
-        props = json.dumps({self.key: self.value})
+        props = json.dumps({self.key: self.value}, ensure_ascii=False)
         return "{name}{props}".format(name=self.type_name, props=props)
 
     @classmethod

rasa/core/interpreter.py

@@ -84,14 +84,14 @@ def _create_entities(parsed_entities, sidx, eidx):
 
     @staticmethod
     def _parse_parameters(
-        entitiy_str: Text, sidx: int, eidx: int, user_input: Text
+        entity_str: Text, sidx: int, eidx: int, user_input: Text
     ) -> List[Dict[Text, Any]]:
-        if entitiy_str is None or not entitiy_str.strip():
+        if entity_str is None or not entity_str.strip():
             # if there is nothing to parse we will directly exit
             return []
 
         try:
-            parsed_entities = json.loads(entitiy_str)
+            parsed_entities = json.loads(entity_str)
             if isinstance(parsed_entities, dict):
                 return RegexInterpreter._create_entities(parsed_entities, sidx, eidx)
             else:
@@ -103,9 +103,9 @@ def _parse_parameters(
         except Exception as e:
             logger.warning(
                 "Invalid to parse arguments in line "
-                "'{}'. Failed to decode parameters"
-                "as a json object. Make sure the intent"
-                "followed by a proper json object. "
+                "'{}'. Failed to decode parameters "
+                "as a json object. Make sure the intent "
+                "is followed by a proper json object. "
                 "Error: {}".format(user_input, e)
             )
             return []

rasa/core/training/interactive.py

@@ -764,7 +764,7 @@ async def _write_stories_to_file(
         for conversation in sub_conversations:
             parsed_events = events.deserialise_events(conversation)
             s = Story.from_events(parsed_events)
-            f.write(s.as_story_string(flat=True) + "\n")
+            f.write("\n" + s.as_story_string(flat=True))
 
 
 async def _write_nlu_to_file(

rasa/core/training/visualization.py

@@ -269,7 +269,7 @@ async def _replace_edge_labels_with_nodes(
             graph.remove_edge(s, e, k)
             graph.add_node(
                 next_id,
-                label=label,
+                label=sanitize(label),
                 shape="rect",
                 style="filled",
                 fillcolor="lightblue",
@@ -300,7 +300,7 @@ def persist_graph(graph, output_file):
         "// { graph-content }", "graph = `{}`".format(expg.to_string()), 1
     )
 
-    with open(output_file, "w") as file:
+    with open(output_file, "w", encoding="utf-8") as file:
         file.write(template)
 
 
@@ -361,7 +361,7 @@ def _create_graph(fontsize: int = 12) -> "networkx.MultiDiGraph":
 
 def sanitize(s):
     if s:
-        return re.sub(r"""[&\\\<\>"'%();+]""", "", s)
+        return re.escape(s)
     else:
         return s
 
@@ -376,8 +376,8 @@ def _add_message_edge(
     """Create an edge based on the user message."""
 
     if message:
-        message_key = sanitize(message.get("intent", {}).get("name", None))
-        message_label = sanitize(message.get("text", None))
+        message_key = message.get("intent", {}).get("name", None)
+        message_label = message.get("text", None)
     else:
         message_key = None
         message_label = None
@@ -464,7 +464,7 @@ async def visualize_neighborhood(
                     next_node_idx,
                     label="  ?  "
                     if not message
-                    else sanitize(message.get("intent", {})).get("name", "  ?  "),
+                    else sanitize(message.get("intent", {}).get("name", "  ?  ")),
                     shape="rect",
                     **{"class": "intent dashed active"}
                 )

rasa/core/utils.py

@@ -341,13 +341,6 @@ def cap_length(s, char_limit=20, append_ellipsis=True):
         return s
 
 
-def write_request_body_to_file(request: Request, path: Text):
-    """Writes the body of `request` to `path`."""
-
-    with open(path, "w+b") as f:
-        f.write(request.body)
-
-
 def extract_args(
     kwargs: Dict[Text, Any], keys_to_extract: Set[Text]
 ) -> Tuple[Dict[Text, Any], Dict[Text, Any]]:

rasa/model.py

@@ -1,12 +1,9 @@
 import glob
-import json
 import logging
 import os
 import shutil
-import tarfile
 import tempfile
-from _md5 import md5
-from typing import Text, Tuple, Union, Optional, List, Dict, Any
+from typing import Text, Tuple, Union, Optional, List, Dict
 
 import yaml.parser
 
@@ -82,48 +79,6 @@ def get_latest_model(model_path: Text = DEFAULT_MODELS_PATH) -> Optional[Text]:
     return max(list_of_files, key=os.path.getctime)
 
 
-def add_evaluation_file_to_model(
-    model_path: Text, payload: Union[Text, Dict[Text, Any]], data_format: Text = "json"
-) -> Text:
-    """Adds NLU data `payload` to zipped model at `model_path`.
-
-    Args:
-        model_path: Path to zipped Rasa Stack model.
-        payload: Json payload to be added to the Rasa Stack model.
-        data_format: NLU data format of `payload` ('json' or 'md').
-
-    Returns:
-        Path of the new archive in a temporary directory.
-    """
-
-    # create temporary directory
-    tmpdir = tempfile.mkdtemp()
-
-    # unpack archive
-    _ = unpack_model(model_path, tmpdir)
-
-    # add model file to folder
-    if data_format == "json":
-        data_path = os.path.join(tmpdir, "data.json")
-        with open(data_path, "w") as f:
-            f.write(json.dumps(payload))
-    elif data_format == "md":
-        data_path = os.path.join(tmpdir, "nlu.md")
-        with open(data_path, "w") as f:
-            f.write(payload)
-    else:
-        raise ValueError("`data_format` needs to be either `md` or `json`.")
-
-    zipped_path = os.path.join(tmpdir, os.path.basename(model_path))
-
-    # re-archive and post
-    with tarfile.open(zipped_path, "w:gz") as tar:
-        for elem in os.scandir(tmpdir):
-            tar.add(elem.path, arcname=elem.name)
-
-    return zipped_path
-
-
 def unpack_model(model_file: Text, working_directory: Optional[Text] = None) -> Text:
     """Unpacks a zipped Rasa model.