Skip to content
Benchmarking tool for various intent and entity classification systems
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
configs/deeppavlov0.0.8
datasets
generated
results
src
tests
.gitignore
README.md
bench.py
docker-compose.yml
requirements.txt

README.md

bench

Benchmarking tool for various intent and entity classification systems.

Installation

Open-source systems (Rasa, DeepPavlov)

Set terminal current directory to the project root (where bench.py is). The docker images can then be build for each Dockerfile having location systems/<system>/Dockerfile where <system> is the folder name of some system using:

docker build -t <system_tag> systems/<system> 

For example docker build -t rasa0.5-mitie0.2 systems/rasa-mitie.

To test the docker file use docker run -it <system_tag>.

To run all the build and tagged Dockers at the same time use

docker-compose up

Packages used in the benchmarks are listed in requirements.txt and can be installed by using pip install -r requirements.txt.

Docker-compose is used to avoid starting various Docker containers from Python. Multiple containers are needed to benchmark systems with different configurations (for example, Rasa MITIE and Rasa spaCy + sklearn). One big issue of starting Docker containers from Python is that Docker requires root privileges.

Cloud services

Watson

Specify Watson API key via environment variable WATSON_USERNAME and WATSON_PASSWORD. For Ubuntu this can be done via changing nano /etc/environment. Validation via printenv <var name (optional)>

DialogFlow

See the DialogFlow v2 API documentation.

TODO: Set project id via environment variable.

You can’t perform that action at this time.