Merge branch 'change_defaults' of github.com:RasaHQ/rasa into change_…

…defaults

changelog/3765.feature.rst

@@ -0,0 +1,18 @@
+Add support for entities with roles and grouping of entities in Rasa NLU.
+
+You can now define a role and/or group label in addition to the entity type for entities.
+Use the role label if an entity can play different roles in your assistant.
+For example, a city can be a destination or a departure city.
+The group label can be used to group multiple entities together.
+For example, you could group different pizza orders, so that you know what toppings goes with which pizza and
+what size which pizza has.
+For more details see :ref:`entities-roles-groups`.
+
+To fill slots from entities with a specific role/group, you need to either use forms or use a custom action.
+We updated the tracker method ``get_latest_entity_values`` to take an optional role/group label.
+If you want to use a form, you can add the specific role/group label of interest to the slot mapping function
+``from_entity`` (see :ref:`forms``).
+
+.. note::
+
+    Composite entities are currently just supported by the :ref:``diet-classifier`` and :ref:``CRFEntityExtractor``.

changelog/5230.bugfix.rst

@@ -1 +1 @@
-Fixed issue where posting to certain callback channel URLs would return a 500 error on successful posts due to invalid response format
+Fixed issue where posting to certain callback channel URLs would return a 500 error on successful posts due to invalid response format.

changelog/5465.feature.rst

@@ -0,0 +1,29 @@
+Update training data format for NLU to support entities with a role or group label.
+
+You can now specify synonyms, roles, and groups of entities using the following data format:
+Markdown:
+
+.. code-block:: none
+
+  [LA]{"entity": "location", "role": "city", "group": "CA", "value": "Los Angeles"}
+
+JSON:
+
+.. code-block:: none
+
+  "entities": [
+      {
+          "start": 10,
+          "end": 12,
+          "value": "Los Angeles",
+          "entity": "location",
+          "role": "city",
+          "group": "CA",
+      }
+  ]
+
+The markdown format ``[LA](location:Los Angeles)`` is deprecated. To update your training data file just
+execute the following command on the terminal of your choice:
+``sed -i .deprecated -E 's/\[(.*)\]\((.*):(.*)\)/\[\1\]\{"entity": "\2", "value": "\3"\}/g' nlu.md``
+
+For more information about the new data format see :ref:`training-data-format`.

changelog/5672.improvement.rst

@@ -0,0 +1,2 @@
+Raise a warning in ``CRFEntityExtractor`` and ``DIETClassifier`` if entities are not correctly annotated in the
+training data, e.g. their start and end values do not match any start and end values of tokens.

data/test/demo-rasa-composite-entities.md

@@ -0,0 +1,56 @@
+## intent:affirm
+- yes
+- yep
+- yeah
+- indeed
+- that's right
+- ok
+- great
+- right, thank you
+- correct
+- great choice
+- sounds really good
+
+## intent:goodbye
+- bye
+- goodbye
+- good bye
+- stop
+- end
+- farewell
+- Bye bye
+- have a good one
+
+## intent:greet
+- hey
+- howdy
+- hey there
+- hello
+- hi
+- good morning
+- good evening
+- dear sir
+
+## intent:chitchat
+- What's your name?
+- What can I call you?
+- How's the weather?
+- Is it too hot outside?
+
+## intent:book_flight
+- i'm looking for a flight
+- I want to book a flight
+- i'm looking for a flight to [Berlin]{"entity": "location", "role": "to"}
+- show me flights from [Amsterdam]{"entity": "location", "role": "from"}
+- show me flights to [London]{"entity": "location", "role": "to"}
+- i am looking for a flight from [SF]{"entity": "location", "value": "San Fransisco", "role": "from"} to [New York]{"entity": "location", "role": "to"}
+- search for flights
+- from [Madrid]{"entity": "location", "role": "from"} to [Munich]{"entity": "location", "role": "to"}
+- any flight to [Liverpool]{"entity": "location", "role": "to"}
+
+## intent:order_pizza
+- i want a [large]{"entity": "size", "group": "1"} pizza with [tomato]{"entity": "topping", "group": "1"} and a [small]{"entity": "size", "group": "2"} pizza with [bacon]{"entity": "topping", "group": "2"}
+- one [large]{"entity": "size", "group": "1"} with [pepperoni]{"entity": "topping", "group": "1"} and a [medium]{"entity": "size", "group": "2"} with [mushrooms]{"entity": "topping", "group": "2"}
+- I would like a [medium]{"entity": "size", "group": "1"} standard pizza and a [medium]{"entity": "size", "group": "2"} pizza with [extra cheese]{"entity": "topping", "group": "2"}
+- [large]{"entity": "size", "group": "1"} with [onions]{"entity": "topping", "group": "1"} and [small]{"entity": "size", "group": "2"} with [olives]{"entity": "topping", "group": "2"}
+- a pizza with [onions]{"entity": "topping", "group": "1"} in [medium]{"entity": "size", "group": "1"} and one with [mushrooms]{"entity": "topping", "group": "2"} in [small]{"entity": "size", "group": "2"} please

docs/core/forms.rst

@@ -142,10 +142,11 @@ Here's an example for the restaurant bot:
 
 The predefined functions work as follows:
 
-- ``self.from_entity(entity=entity_name, intent=intent_name)``
+- ``self.from_entity(entity=entity_name, intent=intent_name, role=role_name, group=group_name)``
   will look for an entity called ``entity_name`` to fill a slot
   ``slot_name`` regardless of user intent if ``intent_name`` is ``None``
-  else only if the users intent is ``intent_name``.
+  else only if the users intent is ``intent_name``. If ``role_name`` and/or ``group_name``
+  are provided, the role/group label of the entity also needs to match the given values.
 - ``self.from_intent(intent=intent_name, value=value)``
   will fill slot ``slot_name`` with ``value`` if user intent is ``intent_name``.
   To make a boolean slot, take a look at the definition of ``outdoor_seating``

docs/nlu/entity-extraction.rst

@@ -31,7 +31,7 @@ Component                  Requires           Model           	        Notes
 The "entity" Object
 ^^^^^^^^^^^^^^^^^^^
 
-After parsing, an entity is returned as a dictionary.  There are two fields that show information
+After parsing, an entity is returned as a dictionary. There are two fields that show information
 about how the pipeline impacted the entities returned: the ``extractor`` field
 of an entity tells you which entity extractor found this particular entity, and
 the ``processors`` field contains the name of components that altered this
@@ -62,7 +62,7 @@ exactly. Instead it will return the trained synonym.
 
     The ``confidence`` will be set by the ``CRFEntityExtractor`` component. The
     ``DucklingHTTPExtractor`` will always return ``1``. The ``SpacyEntityExtractor`` extractor
-    and ``DIETClassifier`` do not provide this information and returns ``null``.
+    and ``DIETClassifier`` do not provide this information and return ``null``.
 
 
 Some extractors, like ``duckling``, may include additional information. For example:
@@ -98,11 +98,93 @@ Custom Entities
 Almost every chatbot and voice app will have some custom entities.
 A restaurant assistant should understand ``chinese`` as a cuisine,
 but to a language-learning assistant it would mean something very different.
-The ``CRFEntityExtractor`` component can learn custom entities in any language, given
+The ``CRFEntityExtractor`` and the ``DIETClassifier`` component can learn custom entities in any language, given
 some training data.
 See :ref:`training-data-format` for details on how to include entities in your training data.
 
 
+.. _entities-roles-groups:
+
+Entities Roles and Groups
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. warning::
+   This feature is experimental.
+   We introduce experimental features to get feedback from our community, so we encourage you to try it out!
+   However, the functionality might be changed or removed in the future.
+   If you have feedback (positive or negative) please share it with us on the `forum <https://forum.rasa.com>`_.
+
+Assigning custom entity labels to words, allow you to define certain concepts in the data.
+For example, we can define what a `city` is:
+
+.. code-block:: none
+
+    I want to fly from [Berlin](city) to [San Francisco](city).
+
+However, sometimes you want to specify entities even further.
+Let's assume we want to build an assistant that should book a flight for us.
+The assistant needs to know which of the two cities in the example above is the departure city and which is the
+destination city.
+``Berlin`` and ``San Francisco`` are still cities, but they play a different role in our example.
+To distinguish between the different roles, you can assign a role label in addition to the entity label.
+
+.. code-block:: none
+
+    - I want to fly from [Berlin]{"entity": "city", "role": "departure"} to [San Francisco]{"entity": "city", "role": "destination"}.
+
+You can also group different entities by specifying a group label next to the entity label.
+The group label can, for example, be used to define different orders.
+In the following example we use the group label to reference what toppings goes with which pizza and
+what size which pizza has.
+
+.. code-block:: none
+
+    Give me a [small]{"entity": "size", "group": "1"} pizza with [mushrooms]{"entity": "topping", "group": "1"} and
+    a [large]{"entity": "size", "group": "2"} [pepperoni]{"entity": "topping", "group": "2"}
+
+See :ref:`training-data-format` for details on how to define entities with roles and groups in your training data.
+
+The entity object returned by the extractor will include the detected role/group label.
+
+.. code-block:: json
+
+    {
+      "text": "Book a flight from Berlin to SF",
+      "intent": "book_flight",
+      "entities": [
+        {
+          "start": 19,
+          "end": 25,
+          "value": "Berlin",
+          "entity": "city",
+          "role": "departure",
+          "extractor": "DIETClassifier",
+        },
+        {
+          "start": 29,
+          "end": 31,
+          "value": "San Francisco",
+          "entity": "city",
+          "role": "destination",
+          "extractor": "DIETClassifier",
+        }
+      ]
+    }
+
+.. note::
+
+    Composite entities are currently only supported by the :ref:``diet-classifier`` and :ref:``CRFEntityExtractor``.
+
+In order to properly train your model with entities that have roles/groups, make sure to include enough training data
+examples for every combination of entity and role/group label.
+Also make sure to have some variations in your training data, so that the model is able to generalize.
+For example, you should not only have example like ``fly FROM x TO y``, but also include examples like
+``fly TO y FROM x``.
+
+To fill slots from entities with a specific role/group, you need to either define a custom slot mappings using
+:ref:`forms` or use :ref:`custom-actions` to extract the corresponding entity directly from the tracker.
+
+
 Extracting Places, Dates, People, Organisations
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^