Simple open-source Rasa command-line add-on that generates training model health reports for your projects.

---

🔍 About

Rasa is the most popular open-source framework for building chat and voice-based AI assistants. The rasa-model-report is a unofficial Rasa add-on to facilitate the work of developers and curators of Rasa chatbots. Rasa provides a lot of valuable data that can be "faceted" and extract different information about the training model. This information makes it possible to discover problems in the training model. The rasa-model-report does just that, it extracts this information to be displayed more clearly in a report. You can see this example.

rasa-model-report is an open-source project.

☕ Donate

You can collaborate with this project donating any amount. Thanks! ☕

📜 Changelog

Changelog can be found here. You can also follow the releases on Github and find planned enhancements for project in the Project Board.

📦 Installation

This module is distributed via Pypi and is required to use Python v3.8 or higher. To install the package, use the command:

pip install rasa-model-report

🚀 Execution

Before anything, is necessary to have the reports generated by the rasa test command. To run the program, use the command:

rasa-model-report

This command must be used in the root of your Rasa project. Otherwise, you can use --path parameter to pass the project path.

Step-by-step

This is the step-by-step guide for using rasa-model-report in your project.

1. Go to the root folder of your Rasa project.
2. Train model on your Rasa project using rasa train command.
3. Run Rasa end-to-end tests using rasa test command.
   • This command will generate some data in json, markdown and image files in result/ directory.
   • This data is needed for rasa-model-report to generate the report.
4. (Optional) If you want to know model NLU rating for each sentence in your project, run your project's Rasa API through the command rasa run --enable-api.
   • When you run rasa-model-report, automatically it will request NLU rating for each sentence. The result will be in the NLU section of the report.
   • If you don't want to use this option, just pass the parameter --disable-nlu or don't run Rasa API (if you don't run Rasa API, rasa-model-report will try to connect, after two tries it will skip this step).
5. Run rasa-model-report in root project.
   • If you haven't installed it, see how to install.
6. The result will be in the model_report.md file generated in the project root folder.

Below, I created this video to show how to use the rasa-model-report v1.0.0. I used the Rasa sample project (from rasa-init command). In this link is the generated report.

example.mp4

Furthermore, I used the tool in Sara - the Rasa Demo Bot. The result you can check here.

🦾 Rasa Version Support

Not every version of Rasa is supported. Check the table below:

Rasa version	Supported
3.X	✅
2.X	✅
1.X	❌
0.X	❌

⚙️ Options

There are parameters that can be used. Available options are below:

--actions-path TEXT     Actions path. (default: actions/ inside Rasa project
                        path)
--disable-nlu           Disable processing NLU sentences. NLU section will
                        not be generated in the report. Required Rasa API.
-e, --exclude LIST      List of utter and actions that will be exclude in
                        the E2E test coverage. Use commas to separate items.
                        Example: utter_greet,utter_goodbye,action_listen
-h, --help              Show this help message.
--model-link TEXT       Model download link. It's only displayed in the
                        report to model download.
--no-images             Generate model report without images.
--output-path TEXT      Report output path. (default: ./)
-p, --path TEXT         Rasa project path. (default: ./)
--precision INTEGER     Score precision. Used to change precision of the
                        model report overview scores. Can vary between 0 and
                        5 (default: 2)
--project-name TEXT     Rasa project name. It's only displayed in the
                        report. (default: My project)
--project-version TEXT  Project version. It's only displayed in the report
                        for project documentation.
--rasa-api TEXT         Rasa API URL. Is needed to create NLU section of
                        report. (default: http://localhost:5005)
--rasa-version TEXT     Rasa version. It's only displayed in the report for
                        project documentation.
-v, --version           Show installed rasa-model-report version.

Usage examples

Some usage examples with parameters:

• Without parameters is usually used at the root of the Rasa project.

  rasa-model-report
  

• When you aren't at the root of the project, use --path parameter.

  rasa-model-report --path path/to/rasa/project
  

• Aren't at the root of the project and without NLU report.

  rasa-model-report --path path/to/rasa/project --disable-nlu
  

• Aren't at the root of the project and change the report output directory.

  rasa-model-report --path path/to/rasa/project --output-path path/to/place/report
  

• Aren't at the root of the project, the actions path isn't at the root project and change the report output directory.

  rasa-model-report --path path/to/rasa/project --output-path path/to/place/report --actions-path path/to/actions/path
  

• If you want exclude some utters and actions from the E2E test coverage.

  rasa-model-report --exclude utter_greet,action_help
  

💻 Development

The instructions for development and contributing are in the CONTRIBUTING.md file.

🐞 Bugs

Please file an issue for bugs, missing documentation, or unexpected behavior.

See open issues

💬 Discussions

Please file an issue to suggest new features. Vote on feature requests. This helps maintainers prioritize what to work on.

See new ideas discussion

❓ Questions

For questions related to using the add-on, please ask the community on Q&A.

See Q&A