Rasa command line interface

[tmbo]

Goal

• intuitive
• short commands (if default configuration)
• reuse existing knowledge
• unify interface for Rasa Core / NLU

General Idea

Assume a rasa project with a default project layout. This means we can assume the location and name of files (e.g. domain.yml is always in the project root and has this name). By looking in default locations for files the user can skip their declaration. If users deviate from the normal project layout, they have to specify the locations manually.

Assumed project layout

.
├── __init__.py
├── actions.py
├── endpoints.yml
├── credentials.yml
├── data
├── domain.yml
├── models
│   ├── core
│   └── nlu
├── config.yml (contains both core and nlu config)

Suggested command line commands:

Creating the initial project (new feature)

• rasa init + interactive guide to create project (e.g. provide project name, etc)

Training

• Training of Core and NLU: rasa train
• Training of the core model: rasa train core
• Training of the nlu model:rasa train nlu
• Interactive Training: rasa interactive (or rasa train interactive?)
• Special case compare polices: rasa train core --config policy1.yml policy2.yml

Validating

• test both: rasa test
• test only core: rasa test core
• test only nlu: rasa test nlu

Run Server

• Start Core with trained NLU model: rasa run
• Run Core server: rasa run core
• Run NLU server: rasa run nlu
• Run action server: rasa run sdk (should we include the sdk or keep that separate?)

Visualize

• Show story paths: rasa show (alternatives: rasa inspect, rasa visualize)

Data

There is currently just one command in this section, but we could have other commands like rasa data validate <file> or rasa data split here in the future

• Convert nlu data: rasa data nlu convert

Interacting with the server

rasa remote

Crucial

• provide credentials: rasa remote login <host> (where / how do we store them)
• select environment (prod / dev / test): rasa remote stages <env>
• set model as active: rasa remote models select <model-id>
• pushing models: rasa remote models push <model>
• downloading models: rasa remote models pull <model>
• push data: rasa remote data push
• pull data: rasa remote data pull

Nice to have

• list models: rasa remote models list
• list users: rasa remote users list
• status : rasa remote status
• logs: rasa remote logs
• show config: rasa remote config

Interacting with local server

rasa local
commands as above for server except

• no login
• pushing / downloading are not needed (set as active should be enough)

Other

• rasa --version
• rasa --help

Parameter Names

• tbd, general idea is to keep them like they are, but unify them between Core and NLU

Current Status

• rasa init
• rasa run
• rasa train
• rasa test
• rasa show
• rasa remote
• rasa local
• rasa data

How to test

https://github.com/RasaHQ/rasa_stack/issues/1#issuecomment-462871671

[wochinge]

@tmbo So this involves specifying the interface as well as implementing it, right?

[tmbo]

Yes, please for both use cases:

• think about which commands we need and how to structure them (I made a first guess on the cli commands on the referenced branch)
• we should then start to implement this api and document it (the implementation should mostly be glueing existing functions together)

[tmbo]

Probably good to discuss a draft with the team / contributors once you have it to check if we are good to go for a first version

[ricwo]

For thoughtless training, can we just have one command rasa train (probably also rasa validate)? This would suit the other single commands like rasa run and make the distinction more seamless.

Re rasa run vs rasa up, I would go for run as that's what we had in core and it will be familiar to many users. I only know up in a docker context, and it might give off the impression that all of this is container-based

[akelad]

Hmm yeah i like having just one command for training, given we don't really want the separation of core/nlu anymore

[wochinge]

@ricwo So rasa train would train both, Core and NLU, at the same time?

[ricwo]

yes, we'd of course still support training them separately, but just as an alias rasa train = rasa train core && rasa train nlu as that's probably what happens under the hood

[amn41]

I like the ideas in here so far!

things not covered yet:

• moving data from md into DB (locally)
• moving trained models from disk into DB (locally)
• syncing data between local rasa x and one running on server
• pushing & pulling trained models to/from remote rasa server

prob need some commands like rasa push model xxx, rasa pull model xxxx, also probably rasa login hostname where you provide password to push and pull stuff.

[EPedrotti]

For the interaction with a remote platform I would like to have something like

• rasa status, shows the platform status and relevant infos (services health, name of core/nlu active models, last trained on, domain, ....)
• rasa list [models | users | actions | intents | ... ]
• rasa config [nlu | core ], shows remote config for nlu/core or both if ommitted
• rasa activate model_id, mark a model as active
• rasa logs service_id
• rasa train [nlu | core] [[remote] [--auto-activate]]

[wochinge]

@EPedrotti Do you think we could separate that a bit from the rest by doing rasa remote + the commands, so e.g. rasa remote status, rasa remote list ?

[EPedrotti]

@wochinge some of this commands make sense only if you are using a remote platform. So maybe it's not necessary to add remote for all remote commands. This will also keep the commands shorter.
On the other hand I don't know if it is better to keep a consistent pattern for "families" of commands, e.g. the remote family.
We could start with a clear separation and see what the users have to say about shortcuts or omitting remote or something else. If we go this way I would try to be as compact as possibile with the commands, without being cryptic, eg -r flag, which is shorter than remote.
Another way is to create more than CLI, AWS does this and I think also Openshift. So we could have one for the local stuff and one for the remote stuff, but i think this works best for huge CLIs, with tons of commands.

That being said it would be nice to have also a rasa init --from-remote url that create the scaffolding but also pulls stuff from the platform. @tmbo do you think it would be possible to download/upload the actions file as well? That would be AWESOME!!!

NOTE: all commands name and flags are just examples

[tmbo]

Some thoughts:

• project layout: there will not be a split between core and nlu, models will probably be model.tar.gz (so no core / nlu subfolders)
• project layout: nlu_config.yml should go into policies.yml (and probably just be config.yml)
• project layout: do we need different files for the different endpoints?
• project layout: putting a module actions in another module actions isn't a great idea, either we go for bot/actions.py or just actions.py (not sure what the python standpoint on src/actions.py is)

Concerning other comments

but just as an alias rasa train = rasa train core && rasa train nlu as that's probably what happens under the hood

We'll need to implement that to be a little more clever (e.g. to avoid retraining nlu if the data for it did not change - because we want people to use rasa train instead of switching to the subcommands because of performance issues)

• moving data from md into DB (locally)

I think the data should live in the files locally, not in the DB (otherwise e.g. rasa train would not work anymore)

• moving trained models from disk into DB (locally)

I think more likely the platform will read them from file

• syncing data between local rasa x and one running on server

I think this is going to be a very interesting and tricky one (but needed)

• pushing & pulling trained models to/from remote rasa server

def

general points:

• I'd rather not introduce commands where we are unsure if they are needed (as it is always easier to add them later, and they will still be callable using python -m syntax - e.g. restore)
• I'd also skip any platform commands that are non essential for the interaction (e.g. upload download model / data is probably important, other parts are nice to have)

[wochinge]

project layout: do we need different files for the different endpoints?

Well, currently Core and NLU both use model: for the model server. If we put this in one file, we'd have to change the implementation of that for at least one of both

project layout: there will not be a split between core and nlu, models will probably be model.tar.gz (so no core / nlu subfolders)

But for the beginning we have to be able to distinguish core from NLU models (e.g. so that rasa run can load the latest of both)

[amn41]

I think the data should live in the files locally, not in the DB (otherwise e.g. rasa train would not work anymore)

how would that work, platform would import the files to sqlite every time it starts up?

  moving trained models from disk into DB (locally) /       I think more likely the platform will read them from file

well we still need some metadata for the models (e.g. is it active, evaluations) in DB in any case. So there is some concept of making platform aware of model;

[tmbo]

@wochinge

Well, currently Core and NLU both use model: for the model server. If we put this in one file, we'd have to change the implementation of that for at least one of both

That is fine, we only need one model server, if the stack is running as one thing.

But for the beginning we have to be able to distinguish core from NLU models (e.g. so that rasa run can load the latest of both)

Not sure what you mean, this should in the combined model case just load the most recent "model pack" which contains nlu & core.

@amn41

how would that work, platform would import the files to sqlite every time it starts up?
I think that makes sense: using the data from the project you start it in. I think we'll need to play around with this a bit, I am also not set on this - I don't think we need to decide on this right now and should just try out which workflows work.

well we still need some metadata for the models (e.g. is it active, evaluations) in DB in any case. So there is some concept of making platform aware of model;
Yes - we'll need to check what we actually need there 👍

[EPedrotti]

I'd also skip any platform commands that are non essential for the interaction (e.g. upload download model / data is probably important, other parts are nice to have)

I think also status and the lists are necessary, otherwise a developer has to switch to the UI to see the status of the platform and to check the logs to see the available custom actions.
If we had only download/upload the CLI would not give enough feedback to the user.

[amn41]

maybe rasa up should have a --use-latest flag or something that automatically makes the most recently trained model active

[wochinge]

maybe --use-latest could even be the default value.

[amn41]

well that makes sense when running locally, but not on the server

[tmbo]

@wochinge the code on the rasa_stack branch https://github.com/RasaHQ/rasa_stack/tree/api uses
https://github.com/RasaHQ/rasa_core/compare/rasa-cli (branch rasa-cli for rasa_core)

To try out the branch, pull the stack and core branch and pip install -e . the stack one. this should install the rasa command for your shell.

[wochinge]

I created a separate issue for the python API: https://github.com/RasaHQ/rasa_stack/issues/4

[wochinge]

• added another command group rasa local to interact with the local server
• updated the remote section

Questions:

• we currently have a parameter url to load stories from a url. Can we just have --stories and check for a http[s]:// prefix?
• we currently use --u / nlu as parameter for the nlu training data. How about renaming that to --messages (as we have --stories for Core?
• @tmbo Why a tar file and not a zip?

[amn41]

1. https prefix seems great - but I guess that'll be a less common use case now that we have rasa data pull
2. I'm not sure I like messages, because it implies you're pulling conversations. maybe utterances ? not sure that's better

[wochinge]

https prefix seems great - but I guess that'll be a less common use case now that we have rasa data pull

👍

2. I'm not sure I like messages, because it implies you're pulling conversations. maybe utterances ? not sure that's better

Utterances sounds too much like bots responses for me.
I'd like to abstract the fact that it is the NLU training data. And the NLU training data are examples of messages. But I agree that this might be confused with conversations. @MetcalfeTom Any ideas?

[akelad]

it is still NLU data though, that is an accurate description of it :D i think maybe it's fine to leave it

[wochinge]

Finally a first prototype is ready.

How to get started:

1. Rasa Core: checkout branch rasa-cli and do pip install -e .
2. Rasa NLU: checkout branch cli and do pip install -e .
3. Rasa Core SDK: checkout branch cli and do pip install -e .
4. Rasa Stack: checkout brach api and do pip install -e .

How to use:
rasa init is probably a could please to start, after that please go crazy

Note:

1. It's a prototype so you will very like encounter bugs. Please tell me then I fix them asap.

[ricwo]

Really cool 👍

• the main init and training work really well in my opinion
• rasa test --help shows training options - are they relevant here?
• can rasa test core/nlu just use whatever model is present when run out of a bot directory? this would do away with having to supply a model path
• if the user does supply a model, is the --model parameter necessary in rasa test? I'd find it more intuitive if it were rasa test nlu model/model.tar as opposed to rasa test nlu --model model/model.tar - especially since that's the only possible argument
• rasa show only supports stories as a positional argument - can this just be the default until there's another rasa show option?
• would it make sense to have just rasa train perform both NLU and Core in one go?
• rasa interactive fails with TypeError: do_interactive_learning() missing 1 required positional argument: 'additional_arguments' - which arguments are required here? could we make that clear in rasa interactive --help?

[wochinge]

rasa test --help shows training options - are they relevant here?

The arguments were relevant, the group name "training" was wrong - fixed

• can rasa test core/nlu just use whatever model is present when run out of a bot directory? this would do away with having to supply a model path

This was an error for the case rasa test core which I fixed. However, I like the syntax rasa test <model> to provide the model 👍

• rasa show only supports stories as a positional argument - can this just be the default until there's another rasa show option

Up for discussion. I decided against but we can easily change that.

• would it make sense to have just rasa train perform both NLU and Core in one go?

One go = Two processes running at the same time? Can do, it might mess up the command line logging for the user.

• rasa interactive fails with TypeError

I somehow don't get this error. Can you please describe what you did? // edit: fixed. Please pull the latest core cli branch

[tmbo]

Really cool I like it - I think this is going into a great direction 🚀

• Startup time: We need to make sure to limit imports, the startup time is way to slow (you can measure using time rasa - for me thats 4 seconds). We can easily improve here by limiting core / nlu module level imports in the cli files (e.g. no from rasa_nlu.model import ... at the top of any file). We should import in the function scope to only import packages when we need them, e.g. when test is called. Right now we are importing the whole universe (spacy, sklearn, tensorflow, keras, ...) which is a bit much.
  I created a PR to scope the imports scoped imports rasa_stack#5. I did a quick check commenting out the imports where possible and the startup time went down to 1.4s which should be fine. There are a couple open Todos that I marked (please take a look at them and resolve them, probably by making sure that these imports don't import keras or anything like that)

• rasa --version: format should be Rasa 0.2.0a1 (no colon, I checked java, git, brew for reference)

• where is the initial_project in "Your bot initial_project is ready to go!" coming from? 😅

• cd my-awsome-bot && rasa train results in an exception (not sure might be related to the import changes I made):

Traceback (most recent call last):
  File "/Users/tmbo/.virtualenvs/py3/bin/rasa", line 11, in <module>
    load_entry_point('rasa', 'console_scripts', 'rasa')()
  File "/Users/tmbo/lastmile/bot-ai/rasa_stack/rasa/__main__.py", line 53, in main
    rasa_core.utils.configure_colored_logging(cmdline_arguments.loglevel)
AttributeError: 'Namespace' object has no attribute 'loglevel'

[tmbo]

I think there should be a little more guidance after the init:

• tell the user that this will create a scaffold project that can be used as a bot template (e.g. link to quickstart)
• we should educate the user on the development workflow after we created the project:
  • change training data -> run rasa train
  • use rasa shell to test the bot

some more thoughts:

• i think we should have a command to run an action server

• properly handle if commands expect a model to be present, but there is none (e.g. running rasa shell in the wrong directory currently results in an exception FileNotFoundError: [Errno 2] No such file or directory: 'models/')

Traceback (most recent call last):
  File "/Users/tmbo/.virtualenvs/py3/bin/rasa", line 11, in <module>
    load_entry_point('rasa', 'console_scripts', 'rasa')()
  File "/Users/tmbo/lastmile/bot-ai/rasa_stack/rasa/__main__.py", line 54, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/Users/tmbo/lastmile/bot-ai/rasa_stack/rasa/cli/shell.py", line 18, in shell
    rasa.cli.run.run(args)
  File "/Users/tmbo/lastmile/bot-ai/rasa_stack/rasa/cli/run.py", line 108, in run
    model_paths = get_model(args.model, subdirectories=True)
  File "/Users/tmbo/lastmile/bot-ai/rasa_stack/rasa/model.py", line 34, in get_model
    return unpack_model(model_path, subdirectories=subdirectories)
  File "/Users/tmbo/lastmile/bot-ai/rasa_stack/rasa/model.py", line 43, in unpack_model
    tar = tarfile.open(model_file)
  File "/Users/tmbo/.virtualenvs/py3/lib/python3.6/tarfile.py", line 1568, in open
    return func(name, "r", fileobj, **kwargs)
  File "/Users/tmbo/.virtualenvs/py3/lib/python3.6/tarfile.py", line 1633, in gzopen
    fileobj = gzip.GzipFile(name, mode + "b", compresslevel, fileobj)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/gzip.py", line 163, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'models/'

• should rasa shell maybe be rasa chat?

[JustinaPetr]

I agree that we should have more guidance at the initial stage for what the user can do (suggesting to check out rasa test --help would be a good start so that users would know what's possible).

Another thing is that this happy path resembles a chatbot a lot now. Which is not a bad thing, but I what I found a bit disappointing is what happens when I say 'no' to questions like 'Do you want me to train an initial model for the bot?' - it responds with 'Great. To train your bot run ...' which feels like another push for the thing I already said I don't want to do and it again leaves me without any other info of what I could do instead.

But in general, it's awesome! So much easier to do things! :)

[wochinge]

Update:

• fixed slow interface due to imports (thanks Tom!)
• tried to make scaffold a bit nicer and more informative
• check if paths are valid
• fixed rasa version
• nicer formatting for rasa --help

Todo:

• rasa shell vs rasa chat
• models param as positional argument (seems a bit tricky with subparsers)
• action server

[howl-anderson]

Really nice work! I have few comments about it:

1. [project layout] merging core, nlu and action into one project is a good option to most user, but if project layout allow user missing some configure or data files (or using empty one) on purpose will be more great. The reason is nlu, core and action services maybe are developed and maintained by different groups of people, so they can work separately and call each other parts by RPC.
2. [new feature] maybe we can provide a command like rasa remote models load <model> to tell rasa load project/model into memory otherwise first request will be very slow. Also, unloading command such as rasa remote models unload <model> can provide to reduce memory usage.
3. [new feature] In the long term, rasa should provide extendability (such plugin) to API and CLI, user can build own's function to control rasa's behavior.

[wochinge]

1. project layout allow user missing some configure or data files (or using empty one) on purpose will be more great.

I think this was a bug with the file validations. Can you please try again? Note that you now have to also install the cli branch from the sdk since I added the command now

2. maybe we can provide a command like rasa remote models load <model> to tell rasa load project/model into memory otherwise first request will be very slow.

This should happen automatically in my opinion= @tmbo

3. In the long term, rasa should provide extendability (such plugin) to API and CLI, user can build own's function to control rasa's behavior.

Mhm, I also thought about the option to define aliases or own default parameters. But you want also define completely new functionality, right?

[howl-anderson]

I think this was a bug with the file validations. Can you please try again? Note that you now have to also install the cli branch from the sdk since I added the command now

I don't use it yet, I will try later

Mhm, I also thought about the option to define aliases or own default parameters. But you want also define completely new functionality, right?

Yes, plugin used to create new function, such as I can create a new CLI/API command/function like rasa remote models sync-from-db which load and unload model according to database which maybe useful to me if I have lots of rasa servers to control.

[tmbo]

@howl-anderson even though I see the benefit of having the possibility to extend this (which you can do, as you can just append subparsers to the parser) I don't think this is very important. The reason is, that I think the downside of using slightly different syntax for custom commands (e.g. python -m myrasa or ./myrasa xyz or make remote models ...) doesn't outweight a plugin infrastructure.

[howl-anderson]

@tmbo

custom commands (e.g. python -m myrasa or ./myrasa xyz or make remote models ...)

it can work locally, but it's hard to execute it thought rasa remote or HTTP API call on remote servers, user need SSH login to the servers, devops tools (such ansible) can be a useful tools for this case but it's not easy to use.

For now, a plugin system is not the most important thing. We can find the best way to do it in the future if user truly want it.

[amn41]

Couple of things to keep track of / clean up.

in interactive mode:

• the prompt is still ? Next user input (Ctr-c to abort): , we should make it the same as in the onboarding experience: Your input ->
• after 'export and quit' rasa will try to persist data to data/nlu.md and data/stories.md, should update those to the new paths

in help menu:
run Run a trained model
this is confusing IMO. I guess what we mean is 'start a rasa server which loads the trained model'?

it's not easy to discover what the correct syntax is for rasa show, not obvious that it should be rasa show stories (same is true for rasa data, just typing that alone should bring up usage instructions, currently you only see the usage line if you actually pass another argument.

• rasa show stories just produces a html file, I guess we should also open it?
• rasa data should also be given the train-test-split functionality, e.g. rasa data split 80 or something

[pandasm]

I would suggest we should give a makefile and give it to the users, it is still unclear how to run the action server with endpoints etc after introducing these commands. are we doing something on it