Skip to content
BUNT is a Bot UNderstanding Testbed
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

BUNT (Bot UNderstanding Testbed)

What is BUNT ?

BUNT is a testbed for NLP engines, typically used in chatbots.

Find out more about the story behind BUNT on this blog post

What BUNT can do for you ?

BUNT can evaluate performance on:

  • languages: French and English are supported. Other languages can be added through modules.
  • corpus: Different default corpus of testing data are provided (small talk, misspellings, …) - other corpus can be added by the user
  • apis: 3 NLP-as-a-service providers are supported:,, and Other providers can be added through plugins

How does BUNT work ?

  • Put your credentials in
LUIS_KEY = 'my_luis_key'
  • Select your settings in
ACTION = 'comparator'
METRICS = ['accuracy', 'error_3_penalized']
    'en': ['mails_en', 'misspellings_en'],
    'fr': ['mails_fr', 'misspellings_fr']
  • Run

How can I add my own NLP service ?

  • Create a class Manager inheriting ApiManager from (See for more details):
class MyAPIManager(ApiManager):
    def __init__(self):
    def fit(self, df_train):
    def predict(self, x_list):
  • Add the name of your API in the settings file (in API_HANDLED)

  • In, add a case with your api (in build_api function)

How can I add my own metric ?

  • Add your metric in score_metric method of
if metric == 'your_metric_name':
    # this example is the accuracy metric
    n_all = n_found + n_fallback + n_error
    return n_found / float(n_all)
  • Add your metric name in in METRICS_HANDLED:
METRICS_HANDLED = ['accuracy',..., 'your_metric_name']

How can I add my own test set ?

  • Create a csv file (tab-separated) containing 2 rows

    The utterances are in the row called sentence and the intents are in the other row called intent (you can see one of the files)

  • Add your file in criteria/language folder.

  • Add the name of your file (without the .csv) in CRITERIA in when you want to use it:

    'en': ['smalltalk_en', ... 'your_english_file'],
    'fr': ['smalltalk_fr', 'mails_fr', 'your_french_file']
You can’t perform that action at this time.