Merge branch 'master' into docs-nlu-components

.github/stale.yml

@@ -5,7 +5,9 @@ daysUntilClose: 7
 # Issues with these labels will never be considered stale
 exemptLabels:
   - "type:enhancement ✨"
+  - "type:enhancement :sparkles:"
   - "type:discussion 👨‍👧‍👦"
+  - "type:discussion :family_man_girl_boy:"
 # Label to use when marking an issue as stale
 staleLabel: stale
 # Comment to post when marking an issue as stale. Set to `false` to disable

CHANGELOG.rst

@@ -17,6 +17,25 @@ This project adheres to `Semantic Versioning`_ starting with version 1.0.
 
 .. towncrier release notes start
 
+[1.8.2] - 2020-03-19
+^^^^^^^^^^^^^^^^^^^^
+
+Bugfixes
+--------
+- `#5438 <https://github.com/rasahq/rasa/issues/5438>`_: Fixed bug when installing rasa with ``poetry``.
+- `#5413 <https://github.com/RasaHQ/rasa/issues/5413>`_: Fixed bug with ``EmbeddingIntentClassifier``, where results
+  weren't the same as in 1.7.x. Fixed by setting weight sparsity to 0.
+
+Improved Documentation
+----------------------
+- `#5404 <https://github.com/rasahq/rasa/issues/5404>`_: Explain how to run commands as ``root`` user in Rasa SDK Docker images since version
+  ``1.8.0``. Since version ``1.8.0`` the Rasa SDK Docker images does not longer run as
+  ``root`` user by default. For commands which require ``root`` user usage, you have to
+  switch back to the ``root`` user in your Docker image as described in
+  :ref:`deploying-your-rasa-assistant_custom-dependencies`.
+- `#5402 <https://github.com/RasaHQ/rasa/issues/5402>`_: Made improvements to Building Assistants tutorial
+
+
 [1.8.1] - 2020-03-06
 ^^^^^^^^^^^^^^^^^^^^
 
@@ -42,7 +61,7 @@ Deprecations and Removals
 
 Features
 --------
-- `#2674 <https://github.com/rasahq/rasa/issues/2674>`_: Add default value ``__other__`` to ``values`` of a ``CategoricalSlot``. 
+- `#2674 <https://github.com/rasahq/rasa/issues/2674>`_: Add default value ``__other__`` to ``values`` of a ``CategoricalSlot``.
 
   All values not mentioned in the list of values of a ``CategoricalSlot``
   will be mapped to ``__other__`` for featurization.
@@ -106,7 +125,7 @@ Improvements
 - `#4653 <https://github.com/rasahq/rasa/issues/4653>`_: Added a new command-line argument ``--init-dir`` to command ``rasa init`` to specify
   the directory in which the project is initialised.
 - `#4682 <https://github.com/rasahq/rasa/issues/4682>`_: Added support to send images with the twilio output channel.
-- `#4817 <https://github.com/rasahq/rasa/issues/4817>`_: Part of Slack sanitization: 
+- `#4817 <https://github.com/rasahq/rasa/issues/4817>`_: Part of Slack sanitization:
   Multiple garbled URL's in a string coming from slack will be converted into actual strings.
   ``Example: health check of <http://eemdb.net|eemdb.net> and <http://eemdb1.net|eemdb1.net> to health check of
   eemdb.net and eemdb1.net``
@@ -138,7 +157,7 @@ Bugfixes
   `TwoStageFallbackPolicy <https://rasa.com/docs/rasa/core/policies/#two-stage-fallback-policy>`_ are used together.
 - `#5201 <https://github.com/rasahq/rasa/issues/5201>`_: Fixed incorrectly raised Error encountered in pipelines with a ``ResponseSelector`` and NLG.
 
-  When NLU training data is split before NLU pipeline comparison, 
+  When NLU training data is split before NLU pipeline comparison,
   NLG responses were not also persisted and therefore training for a pipeline including the ``ResponseSelector`` would fail.
 
   NLG responses are now persisted along with NLU data to a ``/train`` directory in the ``run_x/xx%_exclusion`` folder.

changelog/4826.improvement.rst

@@ -0,0 +1,4 @@
+Add full retrieval intent name to message data
+``ResponseSelector`` will now add the full retrieval intent name
+e.g. ``faq/which_version`` to the prediction, making it accessible
+from the tracker.

docs/api/event-brokers.rst

@@ -8,12 +8,22 @@ Event Brokers
 
 .. edit-link::
 
-Rasa Core allows you to stream events to a message broker. The event broker
-emits events into the event queue. It becomes part of the ``TrackerStore``
-which you use when starting an ``Agent`` or launch ``rasa.core.run``.
+An event broker allows you to connect your running assistant to other services that process the data coming 
+in from conversations. For example, you could `connect your live assistant to 
+Rasa X <https://rasa.com/docs/rasa-x/installation-and-setup/existing-deployment/>`_
+to review and annotate conversations or forward messages to an external analytics
+service. The event broker publishes messages to a message streaming service, 
+also known as a message broker, to forward Rasa :ref:`events` from the Rasa server to other services.
+
+.. contents::
+   :local:
+   :depth: 1
+
+Format
+------
 
 All events are streamed to the broker as serialised dictionaries every time
-the tracker updates it state. An example event emitted from the ``default``
+the tracker updates its state. An example event emitted from the ``default``
 tracker looks like this:
 
 .. code-block:: json
@@ -23,17 +33,15 @@ tracker looks like this:
         "timestamp": 1528402837.617099,
         "event": "bot",
         "text": "what your bot said",
-        "data": "some data"
+        "data": "some data about e.g. attachments"
+        "metadata" {
+              "a key": "a value",
+         }
     }
 
 The ``event`` field takes the event's ``type_name`` (for more on event
 types, check out the :ref:`events` docs).
 
-Rasa enables three possible broker types:
-
-- `Pika Event Broker`_
-- `Kafka Event Broker`_
-- `SQL Event Broker`_
 
 .. _event-brokers-pika:
 
@@ -44,21 +52,19 @@ The example implementation we're going to show you here uses
 `Pika <https://pika.readthedocs.io>`_ , the Python client library for
 `RabbitMQ <https://www.rabbitmq.com>`_.
 
+.. contents::
+   :local:
+
 Adding a Pika Event Broker Using the Endpoint Configuration
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-You can use an endpoint configuration file to instruct Rasa Core to stream
-all events to your event broker. To do so, add the following section to your
-endpoint configuration, e.g. ``endpoints.yml``:
+You can instruct Rasa to stream all events to your Pika event broker by adding an ``event_broker`` section to your
+``endpoints.yml``:
 
 .. literalinclude:: ../../data/test_endpoints/event_brokers/pika_endpoint.yml
 
-Then instruct Rasa Core to use the endpoint configuration and Pika producer by adding
-``--endpoints <path to your endpoint configuration`` as following example:
+Rasa will automatically start streaming events when you restart the Rasa server.
 
-.. code-block:: shell
-
-    rasa run -m models --endpoints endpoints.yml
 
 Adding a Pika Event Broker in Python
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -68,14 +74,14 @@ Here is how you add it using Python code:
 .. code-block:: python
 
     from rasa.core.brokers.pika import PikaEventBroker
-    from rasa_platform.core.tracker_store import InMemoryTrackerStore
+    from rasa.core.tracker_store import InMemoryTrackerStore
 
     pika_broker = PikaEventBroker('localhost',
                                   'username',
                                   'password',
                                   queues=['rasa_events'])
 
-    tracker_store = InMemoryTrackerStore(db=db, event_broker=pika_broker)
+    tracker_store = InMemoryTrackerStore(domain=domain, event_broker=pika_broker)
 
 
 Implementing a Pika Event Consumer
@@ -121,35 +127,20 @@ It is possible to use `Kafka <https://kafka.apache.org/>`_ as main broker for yo
 events. In this example we are going to use the `python-kafka <https://kafka-python
 .readthedocs.io/en/master/usage.html>`_ library, a Kafka client written in Python.
 
-.. note::
-
-  In order to use the Kafka event broker, ``rasa`` has to be installed with the
-  ``kafka`` option:
-
-  .. code-block:: bash
-
-    $ pip install rasa[kafka]
-
+.. contents::
+   :local:
 
 Adding a Kafka Event Broker Using the Endpoint Configuration
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-As for the other brokers, you can use an endpoint configuration file to instruct Rasa Core to stream
-all events to this event broker. To do it, add the following section to your
-endpoint configuration.
-
-Pass the ``endpoints.yml`` file as argument with ``--endpoints <path to your endpoint configuration>``
-when running Rasa, as following example:
-
-.. code-block:: shell
-
-    rasa run -m models --endpoints endpoints.yml
+You can instruct Rasa to stream all events to your Kafka event broker by adding an ``event_broker`` section to your
+``endpoints.yml``.
 
 Using ``SASL_PLAINTEXT`` protocol the endpoints file must have the following entries:
 
 .. literalinclude:: ../../data/test_endpoints/event_brokers/kafka_plaintext_endpoint.yml
 
-In the case of using SSL protocol the endpoints file must looks like:
+If using SSL protocol, the endpoints file should look like:
 
 .. literalinclude:: ../../data/test_endpoints/event_brokers/kafka_ssl_endpoint.yml
 
@@ -166,7 +157,7 @@ The code below shows an example on how to instantiate a Kafka producer in you sc
     kafka_broker = KafkaEventBroker(host='localhost:9092',
                                     topic='rasa_events')
 
-    tracker_store = InMemoryTrackerStore(event_broker=kafka_broker)
+    tracker_store = InMemoryTrackerStore(domain=domain, event_broker=kafka_broker)
 
 
 The host variable can be either a list of brokers adresses or a single one.
@@ -185,10 +176,10 @@ list of strings. e.g.:
                                           'kafka_broker_3:9092'],
                                     topic='rasa_events')
 
-Authentication and authorization
+Authentication and Authorization
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Rasa Core's Kafka producer accepts two types of security protocols - ``SASL_PLAINTEXT`` and ``SSL``.
+Rasa's Kafka producer accepts two types of security protocols - ``SASL_PLAINTEXT`` and ``SSL``.
 
 For development environment, or if the brokers servers and clients are located
 into the same machine, you can use simple authentication with ``SASL_PLAINTEXT``.
@@ -231,7 +222,7 @@ and inter-broker connections to prevent man-in-the-middle attacks.
 Implementing a Kafka Event Consumer
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The parameters used to create a Kafka consumer is the same used on the producer creation,
+The parameters used to create a Kafka consumer are the same used on the producer creation,
 according to the security protocol being used. The following implementation shows an example:
 
 .. code-block:: python
@@ -265,9 +256,8 @@ and PostgreSQL databases, to see other options, please see the
 Adding a SQL Event Broker Using the Endpoint Configuration
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-You can use the endpoint configuration file to instruct Rasa to save
-all events to your SQL event broker. To do so, add a ``event_broker`` section to your
-endpoint configuration, e.g. ``endpoints.yml``. For example, a valid SQLite configuration
+To instruct Rasa to save all events to your SQL event broker, add an ``event_broker`` section to your
+``endpoints.yml``. For example, a valid SQLite configuration
 could look like the following:
 
 .. code-block:: yaml

docs/api/tensorflow_usage.rst

@@ -14,19 +14,33 @@ All configuration options are specified using environment variables as shown in
 Optimizing CPU Performance
 --------------------------
 
+.. note::
+    We recommend that you configure these options only if you are an advanced TensorFlow user and understand the 
+    implementation of the machine learning components in your pipeline. These options affect how operations are carried 
+    out under the hood in Tensorflow. Leaving them at their default values is fine.
+
+Depending on the TensorFlow operations a NLU component or Core policy uses, you can leverage multi-core CPU
+parallelism by tuning these options.
+
 Parallelizing One Operation
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Set ``TF_INTRA_OP_PARALLELISM_THREADS`` as an environment variable to specify the maximum number of threads that can be used
-to parallelize the execution of one operation. If left unspecified, this value defaults to ``0`` which means TensorFlow should
-pick an appropriate value depending on the system configuration.
+to parallelize the execution of one operation. For example, operations like ``tf.matmul()`` and ``tf.reduce_sum`` can be executed
+on multiple threads running in parallel. The default value for this variable is ``0`` which means TensorFlow would
+allocate one thread per CPU core.
 
 Parallelizing Multiple Operations
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Set ``TF_INTER_OP_PARALLELISM_THREADS`` as an environment variable to specify the maximum number of threads that can be used
-to parallelize the execution of multiple **non-blocking** operations. If left unspecified, this value defaults to ``0``
-which means TensorFlow should pick an appropriate value depending on the system configuration.
+to parallelize the execution of multiple **non-blocking** operations. These would include operations that do not have a
+directed path between them in the TensorFlow graph. In other words, the computation of one operation does not affect the
+computation of the other operation. The default value for this variable is ``0`` which means TensorFlow would allocate one thread per CPU core.
+
+To understand more about how these two options differ from each other, refer to this
+`stackoverflow thread <https://stackoverflow.com/a/41233901/3001665>`_.
+
 
 Optimizing GPU Performance
 --------------------------
@@ -35,18 +49,13 @@ Limiting GPU Memory Growth
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 TensorFlow by default blocks all the available GPU memory for the running process. This can be limiting if you are running
-multiple TensorFlow processes and want to distribute memory across them. To prevent this,
-set the environment variable ``TF_FORCE_GPU_ALLOW_GROWTH`` to ``True``.
+multiple TensorFlow processes and want to distribute memory across them. To prevent Rasa Open Source from blocking all
+of the available GPU memory, set the environment variable ``TF_FORCE_GPU_ALLOW_GROWTH`` to ``True``.
 
 Restricting Absolute GPU Memory Available
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Often, a developer wants to limit the absolute amount of GPU memory that can be used by a process.
-
-For example, you may have two visible GPUs(``GPU:0`` and ``GPU:1``) and you want to allocate 1024 MB from the first GPU
-and 2048 MB from the second GPU.
-You can do so by setting an environment variable as ``TF_GPU_MEMORY_ALLOC="0:1024, 1:2048"``.
+You may want to limit the absolute amount of GPU memory that can be used by a Rasa Open Source process.
 
-Another scenario can be where you have access to 2 GPUs(``GPU:0`` and ``GPU:1``) but you would like to use only the second
-GPU.
-``TF_GPU_MEMORY_ALLOC="1:2048"`` would make 2048 MB of memory available from GPU 1.
+For example, say you have two visible GPUs(``GPU:0`` and ``GPU:1``) and you want to allocate 1024 MB from the first GPU
+and 2048 MB from the second GPU. You can do this by setting the environment variable ``TF_GPU_MEMORY_ALLOC`` to ``"0:1024, 1:2048"``.

docs/migration-guide.rst

@@ -92,11 +92,12 @@ General
     - name: DIETClassifier
       hidden_layers_sizes:
         text: [256, 128]
+      number_of_transformer_layers: 0
+      weight_sparsity: 0
       intent_classification: True
       entity_recognition: False
       use_masked_language_model: False
       BILOU_flag: False
-      number_of_transformer_layers: 0
       # ... any other parameters
 
   See :ref:`DIETClassifier <diet-classifier>` for more information about the new component.