Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?


Failed to load latest commit information.
Latest commit message
Commit time

SimpleTOD: A Simple Language Model for Task-Oriented Dialogue

Authors: Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher


SimpleTOD single turn


Task-oriented dialogue (TOD) systems accomplish a goal described by a user in natural language. They often use a pipeline approach. Such approach requires natural language understanding (NLU) for belief state tracking, dialogue management (DM) for deciding which actions to take based on those beliefs, and natural language generation (NLG) for generating responses.

We propose recasting task-oriented dialogue as a simple, causal (unidirectional) language modeling task. We show that such an approach can solve all the sub-tasks in a unified way using multi-task maximum likelihood training. The proposed Simple Task-Oriented Dialogue (SimpleTOD) approach enables modeling of the inherent dependencies between the sub-tasks of task-oriented dialogue, by optimizing for all tasks in an end-to-end manner.

Paper link:

Blog link:

Table of Contents


The package general requirements are

  • Python >= 3.6
  • Pytorch >= 1.2 (installation instructions here)
  • Transformers >= 2.5.1 (installation instructions here)

1- The package can be installed by running the following command.

pip install -r requirements.txt

2- Running inside docker container

docker build -t <image_name>:<tag> -f Dockerfile


This section explains steps to preprocess MultiWOZ dataset and training the model.


It includes downloading MultiWOZ dataset, performing delexicaliztion, and creating dataset for language model

Each dialogue turn will be represented as a sequence, which contains previous user/system turns, belief, action, and delexicalized response

<|endoftext|> <|context|> <|user|> i am looking for a college type attraction . <|system|> there are 18 colleges i have found , would you prefer 1 in town centre or in the west ? <|user|> i would like to visit on in town centre please . <|system|> sure , we have thirteen options , 10 of which are free . may i suggest king s college , or hughes hall ? <|user|> okay , may i have their postcode , entrance fee , and phone number ?<|endofcontext|> 
<|belief|> attraction type college , attraction name kings college|hughes hall , attraction area centre <|endofbelief|> 
<|action|> attraction inform name , attraction inform fee , attraction inform post , attraction inform phone <|endofaction|> 
<|response|> sure , the post code to [attraction_name] is [attraction_postcode] , the entrance fee is free , and phone number [attraction_phone] <|endofresponse|> <|endoftext|>

DST training:

training the model for predicting belief states. $GPU gpt2 $GPT2_TYPE $BATCH

For this task, we include none slot values in the sequence. We observed that this will improve SimpleTOD performance on DST by reducing false positive rates.

<|endoftext|> <|context|> <|user|> am looking for a place to to stay that has cheap price range it should be in a type of hotel <|endofcontext|> 
<|belief|> hotel name not mentioned , hotel area not mentioned , hotel parking not mentioned , hotel pricerange cheap , hotel stars not mentioned , hotel internet not mentioned , hotel type hotel <|endofbelief|> <|endoftext|>

End-to-End training:

In this step, we train SimpleTOD on the sequence of context+belief+action+delex response. Compared to DST task, we do not include none slot values, because of the sequence length limitaiton od GPT2. $GPU gpt2 $GPT2_TYPE $BATCH


This script will generate SimpeTOD belief/action/responses. Generation is based on each dialogue, where it create context for each turn and save the generated belief, action, and responses for the dialogue.


It will save the model output in a json file MODEL_OUTPUT which contains all dialogues with groundtruth user and system responses as well.

  • In order to use DB search during generation, set --use_db_search (this will use oracle DB search results)
  • In order to use DB search dynamically, set --use_db_search and --use_dynamic_db
  • To use oracle belief and actions, simple set --use_oracle_belief and --use_oracle_action


MultiWOZ evaluation contains two part, Dialogue State Tracking (DST) and End-to-End.

DST evaluation

In order to compute joint accuracy, simply run the following script using the generated MODEL_OUTPUT file. it will use the generated belief states to compute the metric. It will compute joint accuracy without any label cleaning.


There are two types of label cleaning that can be used to compute joint accuracy.

  • To use default lable cleaning suggested by MultiWOZ author, please set --default_cleaning (for more details, please refer to MultiWOZ FAQ.5)
  • We found other type of noisy annotation. Please refer to the paper for more details different types of noisy annotations. Here, we provide an option to compute joint accuracy by fixing Type 2 noisy annotation (where one or more slots are not labeled in some turns.) by setting --type2_cleaning
  • The complete list of Type 2 noisy annotations is here. For more details on noisy annotation on MultiWOZ dataset, please refer to the paper

End-to-End evaluation

In order to compute inform/success/BLEU, simply run the following script. It will load generated belief states and responses, and computes the metrics.



In order to test the model in real conversation with human, we have provided a simple script where user can input text in a multi turn setting, and see the responses from SimpleTOD. It will generate lexicalized responses and belief states at each turn. For more information, please read the blog.



  title={A simple language model for task-oriented dialogue},
  author={Hosseini-Asl, Ehsan and McCann, Bryan and Wu, Chien-Sheng and Yavuz, Semih and Socher, Richard},
  journal={arXiv preprint arXiv:2005.00796},


The code is released under the BSD-3 License - see LICENSE for details


Official repository for "SimpleTOD: A Simple Language Model for Task-Oriented Dialogue"



Code of conduct

Security policy





No releases published


No packages published