add migration/changelog

CHANGELOG.rst

@@ -66,10 +66,16 @@ Changed
 - forms were completely reworked, see changelog in ``rasa_core_sdk``
 - state featurization if some form is active changed
 - ``Domain`` raises ``InvalidDomain`` exception
+- interactive learning is now started with rasa_core.train interactive
+- passing a policy config file to train a model is now required
+- flags for output of evaluate script have been merged to one flag ``--output``
+  where you provide a folder where any output from the script should be stored
 
 Removed
 -------
 - removed graphviz dependency
+- policy config related flags in training script (see migration guide)
+
 
 Fixed
 -----

docs/evaluation.rst

@@ -19,17 +19,17 @@ by using the evaluate script:
 
 .. code-block:: bash
 
-    $ python -m rasa_core.evaluate default -d models/dialogue \
-      -s test_stories.md -o matrix.pdf --failed failed_stories.md
+    $ python -m rasa_core.evaluate -d models/dialogue \
+      -s test_stories.md -o results
 
 
-This will print the failed stories to ``failed_stories.md``.
+This will print the failed stories to ``results/failed_stories.md``.
 We count any story as `failed` if at least one of the actions
 was predicted incorrectly.
 
 In addition, this will save a confusion matrix to a file called
-``matrix.pdf``. The confusion matrix shows, for each action in your
-domain, how often that action was predicted, and how often an
+``results/story_confmat.pdf``. The confusion matrix shows, for each action in 
+your domain, how often that action was predicted, and how often an
 incorrect action was predicted instead.
 
 The full list of options for the script is:

docs/interactive_learning.rst

@@ -17,7 +17,7 @@ Some people call this `Software 2.0 <https://medium.com/@karpathy/software-2-0-a
 Load up an existing bot
 ^^^^^^^^^^^^^^^^^^^^^^^
 
-We have created some initial stories, and now want to improve our bot 
+We have created some initial stories, and now want to improve our bot
 by providing feedback on mistakes it makes.
 
 Run the following command to start interactive learning:
@@ -27,15 +27,15 @@ Run the following command to start interactive learning:
    python -m rasa_core_sdk.endpoint --actions actions&
 
    python -m rasa_core.train \
-     --interactive -o models/dialogue \
+     interactive -o models/dialogue \
      -d domain.yml -s stories.md \
      --nlu models/current/nlu \
      --endpoints endpoints.yml
 
 The first command starts the action server (see :ref:`customactions`).
 
 The second command starts the bot in interactive mode.
-In interactive mode, the bot will ask you to confirm every prediction 
+In interactive mode, the bot will ask you to confirm every prediction
 made by NLU and Core before proceeding.
 Here's an example:
 
@@ -62,14 +62,14 @@ Here's an example:
     ? The bot wants to run 'utter_greet', correct?  (Y/n)
 
 
-The chat history and slot values are printed to the screen, which 
-should be all the information your need to decide what the correct 
+The chat history and slot values are printed to the screen, which
+should be all the information your need to decide what the correct
 next action is.
 
 In this case, the bot chose the
 right action (``utter_greet``), so we type ``y``.
 Then we type ``y`` again, because ``action_listen`` is the correct
-action after greeting. We continue this loop, chatting with the bot, 
+action after greeting. We continue this loop, chatting with the bot,
 until the bot chooses the wrong action.
 
 Providing feedback on errors
@@ -138,8 +138,8 @@ reviews!) so we select that action.
 
 Now we can keep talking to the bot for as long as we like to create a longer
 conversation. At any point you can press ``Ctrl-C`` and the bot will
-provide you with exit options. You can write your newly-created stories and NLU 
-data to files. You can also go back a step if you made a mistake when providing 
+provide you with exit options. You can write your newly-created stories and NLU
+data to files. You can also go back a step if you made a mistake when providing
 feedback.
 
 Make sure to combine the dumped stories and NLU examples with your original
@@ -165,20 +165,20 @@ script.
 Interactive Learning with Forms
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-If you're using a FormAction, there are some additional things to keep in mind 
+If you're using a FormAction, there are some additional things to keep in mind
 when using interactive learning.
 
 The ``form:`` prefix
 ~~~~~~~~~~~~~~~~~~~~
 
 The form logic is described by your ``FormAction`` class, and not by the stories.
-The machine learning policies should not have to learn this behavior, and should 
-not get confused if you later change your form action, for example by adding or 
-removing a required slot. 
-When you user interactive learning to generate stories containing a form, 
-the conversation steps handled by the form 
-get a :code:`form:` prefix. This tells Rasa Core to ignore these steps when training 
-your other policies. There is nothing special you have to do here, all of the form's 
+The machine learning policies should not have to learn this behavior, and should
+not get confused if you later change your form action, for example by adding or
+removing a required slot.
+When you user interactive learning to generate stories containing a form,
+the conversation steps handled by the form
+get a :code:`form:` prefix. This tells Rasa Core to ignore these steps when training
+your other policies. There is nothing special you have to do here, all of the form's
 happy paths are still covered by the basic story given in :ref:`section_form_basics`.
 
 Here is an example:
@@ -205,10 +205,10 @@ Here is an example:
 Input validation
 ~~~~~~~~~~~~~~~~
 
-Every time the user responds with something *other* than the requested slot or 
-any of the required slots, 
-you will be asked whether you want the form action to try and extract a slot 
-from the user's message when returning to the form. This is best explained with 
+Every time the user responds with something *other* than the requested slot or
+any of the required slots,
+you will be asked whether you want the form action to try and extract a slot
+from the user's message when returning to the form. This is best explained with
 and example:
 
 .. code-block:: text
@@ -241,7 +241,7 @@ and example:
     ? Should 'restaurant_form' validate user input to fill the slot 'outdoor_seating'?  (Y/n)
 
 Here the user asked to stop the form, and the bot asks the user whether they're sure
-they don't want to continue. The user says they want to continue (the ``/affirm`` intent). 
+they don't want to continue. The user says they want to continue (the ``/affirm`` intent).
 Here ``outdoor_seating`` has a ``from_intent`` slot mapping (mapping
 the ``/affirm`` intent to ``True``), so this user input could be used to fill
 that slot. However, in this case the user is just responding to the
@@ -260,7 +260,7 @@ should not be validated. The bot will then continue to ask for the
 
     **WARNING: FormPolicy predicted no form validation based on previous training
     stories. Make sure to remove contradictory stories from training data**
-    
+
     Once you've removed that story, you can press enter and continue with
     interactive learning
 

docs/migrations.rst

@@ -19,6 +19,26 @@ how you can migrate from one version to another.
     before updating. Please make sure to
     **retrain your models when switching to this version**.
 
+Train script
+~~~~~~~~~~~~
+
+- You **must** pass a policy config flag with ``-c/--config`` now when training
+  a model, see :ref:`policy_file`. There is a default config file ``default_config.yml``
+  in the Github repo
+- Interactive learning is now started with ``python -m rasa_core.train interactive``
+  rather than the `--interactive` flag
+- All policy configuration related flags have been removed (--epochs,
+  --max_history, --validation_split, --batch_size, --nlu_threshold, --core_threshold,
+  --fallback_action_name), specify these in the policy config file instead,
+  see :ref:`policy_file`
+
+Evaluation script
+~~~~~~~~~~~~~~~~
+
+- The ``--output`` flag now takes one argument: the name of the folder any files
+  generated from the script should be written to
+- The ``--failed`` flag was removed, as this is part of the ``--output`` flag now
+
 Forms
 ~~~~~
 
@@ -310,6 +330,3 @@ There have been some API changes to classes and methods:
 
 
 .. include:: feedback.inc
-
-
-

docs/policies.rst

@@ -18,7 +18,7 @@ You can run training from the command line like in the :ref:`quickstart`:
 .. code-block:: bash
 
    python -m rasa_core.train -d domain.yml -s data/stories.md \
-     -o models/current/dialogue --epochs 200
+     -o models/current/dialogue -c default_config.yml
 
 Or by creating an agent and running the train method yourself:
 
@@ -66,8 +66,8 @@ One important hyperparameter for Rasa Core policies is the ``max_history``.
 This controls how much dialogue history the model looks at to decide which
 action to take next.
 
-You can set the ``max_history`` using the training script's ``--history``
-flag or by passing it to your policy's ``Featurizer``.
+You can set the ``max_history`` by passing it to your policy's ``Featurizer``
+in the policy configuration yaml file.
 
 .. note::
 
@@ -102,7 +102,7 @@ slot. Slot information is always available for every featurizer.
 Training Script Options
 ^^^^^^^^^^^^^^^^^^^^^^^
 
-.. program-output:: python -m rasa_core.train -h
+.. program-output:: python -m rasa_core.train default -h
 
 
 
@@ -122,15 +122,20 @@ highest confidence will be used.
 Configuring polices using a configuration file
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-You can set the policies you would like the Core model to use in a YAML file.
+If you are using the training script, you must set the policies you would like
+the Core model to use in a YAML file.
 
 For example:
 
 .. code-block:: yaml
 
   policies:
     - name: "KerasPolicy"
-      max_history: 5
+      featurizer:
+      - name: MaxHistoryTrackerFeaturizer
+        max_history: 5
+        state_featurizer:
+          - name: BinarySingleStateFeaturizer
     - name: "MemoizationPolicy"
       max_history: 5
     - name: "FallbackPolicy"
@@ -141,9 +146,8 @@ For example:
       arg1: "..."
 
 Pass the YAML file's name to the train script using the ``--config``
-argument (or just ``-c``). If no config.yaml is given, the policies
-default to
-``[KerasPolicy(), MemoizationPolicy(), FallbackPolicy(), FormPolicy()]``.
+argument (or just ``-c``). There is a default config file you can use in the
+github repository called ``default_config.yml``
 
 .. note::
 
@@ -377,5 +381,3 @@ It is recommended to use
 
 
 .. include:: feedback.inc
-
-

examples/formbot/Makefile

@@ -25,7 +25,7 @@ run:
 	make run-core
 
 train-interactive:
-	python -m rasa_core.train --interactive -s data/stories.md -d domain.yml -o models/dialogue --debug --endpoints endpoints.yml
+	python -m rasa_core.train interactive -s data/stories.md -d domain.yml -o models/dialogue --debug --endpoints endpoints.yml
 
 visualize:
 	python -m rasa_core.visualize -s data/stories.md -d domain.yml -o story_graph.png

examples/moodbot/Makefile

@@ -11,7 +11,7 @@ train-nlu:
 	       --data ./data/nlu.md --path models/ --project nlu
 
 train-core:
-	python -m rasa_core.train default -s data/stories.md -d domain.yml -o models/dialogue -c ../../default_config.yml
+	python -m rasa_core.train -s data/stories.md -d domain.yml -o models/dialogue -c ../../default_config.yml
 
 run-fb:
 	python -m rasa_core.run -d models/dialogue -u models/nlu/current -p 5002 -c facebook --credentials credentials.yml

rasa_core/evaluate.py

@@ -28,7 +28,8 @@
 from rasa_core.policies import SimplePolicyEnsemble
 from rasa_core.trackers import DialogueStateTracker
 from rasa_core.training.generator import TrainingDataGenerator
-from rasa_core.utils import AvailableEndpoints, pad_list_to_size
+from rasa_core.utils import (AvailableEndpoints, pad_list_to_size,
+                             set_default_subparser)
 
 from rasa_nlu import utils as nlu_utils
 from rasa_nlu.evaluate import plot_confusion_matrix, get_evaluation_metrics
@@ -636,6 +637,7 @@ def plot_curve(output, no_stories, ax=None, **kwargs):
 if __name__ == '__main__':
     # Running as standalone python application
     arg_parser = create_argument_parser()
+    set_default_subparser(arg_parser, 'default')
     cmdline_args = arg_parser.parse_args()
 
     logging.basicConfig(level=cmdline_args.loglevel)