A conversational system has four components:
- Input/Output modules - These mainly comprise of the Automated Speech Recognition (ASR) engine and the Text To Speech (TTS) engine.
- Natural Language Understanding module (NLU) - This part of the conversational system is responsible for breaking down human langugage into machine readable form. In simple terms, the NLU tries to break down the user utterances into intents and entities.
- Natural Language Generation module (NLG) - The NLG module is concerned with the generation of human readable text from some data source. The NLG is responsible for not just producing syntactically correct sentences, but it also needs to make sure the generated text makes sense.
- Dialog Management System - This component of the conversational system is responsible for "guiding" the flow of the conversation. It uses the input from the NLU, decides how the conversation should proceed and accordingly directs the NLG what to output to the user.
- Instead of designing separate models for intent and entity detection, we decided to create a multi-task model that can be jointly trained and optimized for both these tasks.
- We have developed a transformer based natural language generation model fit for task-oriented conversational systems. The transformer used is OpenAI’s GPT which has shown incredible capacity for text generation in recent times.
- We use GPT, which has till now only been used for generation purposes, to also do the work of the NLU module i.e. a transformer is used to extract entities and intents from the input utterances. (novel approach)
- The NLU and NLG components are fused together into one conversational system. The dialog management system is completely discarded. The combination of NLU+NLG modules we have developed is robust enough to hold meaningful task-oriented conversations with the user. (novel approach)
- Finally, as a sample application, we have developed a restaurant information conversational system which can chat with the user in real time, and which confirms that our approach to conversational systems is indeed a promising one and worthy of future research.
- Intent extraction component - This is a bidirectional LSTM which takes in the input embeddings and outputs the intent of the utterance using a linear layer.
- Entity extraction component - This is a bidirectional LSTM with a CRF layer on top. The output is of equal length as the input with an entity type for each input word.
- Embedding component - We use ELMo embeddings to convert the input words into machine understandable form. ELMo embeddings are context dependent and result in better performace when compared with our baseline Word2Vec and FastText embedding models.
Overall Model - The ELMo embedding component forms the base of the multi-task NLU model. The Intent and Entity recognition components sit on top of the ELMo component. The final BiLSTM layer of the ELMo module is trained as a shared BiLSTM between the Intent and Entity recognition components.
Metrics -We calculate two metrics on the test set to measure the performance of our NLU model:
Intent Classification Accuracy (ICA) = 96.8 %
Entity F1 Score = 97.4 %
We have used the GPT for the generative needs of our conversational system. GPT consists of a stack of 12 transformer decoder layers stacked one above the other. The GPT, which is suited for tasks like passage generation, is adapted to a conversational setting by making a few tweaks to the structure of data fed into the model.
We have taken inspiration from two recent works on how to adapt GPT to a conversational setting.
The TransferTransfo framework is suitable for open domain conversations. We, however, are building a task-oriented conversational system and hence need to make some adjustments to the structure of TransferTransfo. The authors of this paper propose a method to convert the TransferTransfo framework into one suitable for task-oriented conversations. The authors concatenate the domain name, the belief state (the filled in slots), the database state (number of slots in the domain) and the conversation history to the input that is sent to the GPT, and the GPT produces the system output based on all this information.
At this point we had a strong foundation to proceed with the creation of our task-oriented restaurant information conversational system. We made two modifications to the model above. The Belief state, being in the beginning of the conversation, is too far away from the current response and hence its effect is diminished on the response. We shift the belief state closer to the current response. Although transformers are position agnostic, the slots accumulated in the beginning of the input text made it difficult for the model to connect particular slots with the corresponding parts of the conversation history and also led to problems when the direction of the conversation was changed (for example, from cheap food to expensive food). This was noticed during our initial testing. After making the changes, the model was able to hold long properly directed conversations with the userand was also able to adapt well to abrupt changes in the conversation.
At this point there arose another problem. How do we fill the slots? The multi-task NLU model developed at the beginning of the project came to our rescue, and we quickly adapted it to our current domain. The NLU model succeeded in predicting the slots with high accuracy. In this way, the NLU and NLG models fused together into a robust task-oriented conversational system.
However, while using combination model worked well, it had a long inference time as the information needed to be passed through two separate neural networks. We came up with a novel way of solving this problem which enabled us to drastically cut down the average inference time from 1.8 seconds to 0.9 seconds on a GeForce MX150.
- Greedy - The token with the highest probability is predicted. This leads to less natural conversations. Another problem is that a high-probability token may be hiding after a low-probability token and may be missed.
- Beam Search - Tries to mitigate this issue by maintaining a beam of several possible sequences that we construct word-by-word. At the end of the process, we select the best sentence among the beams.
- Random Sampling - This randomly samples from all the tokens based on their probability. So, every token has a chance of getting predicted, but the ones with higher probability are more probable.
- Top-K Sampling - This takes the top K number of tokens which have the highest probability and samples from them.
- Nucleus Sampling (Top-P Sampling) - This takes the smallest set of the top V tokens, the sum of whose probabilities add up to P.
As evident from the graph, nucleus sampling is the best approach among the five and leads to utterances very similar to human language. We use this approach for decoding the next predicted token and use P = 0.85.
Dataset - We created a custom dataset by combining the MultiWoz and Dstc2 datasets. The dataset has 2212 training samples, 706 validation samples and 816 testing samples.
Intents - Two major intents with several sub-intents: Request and Inform.
Entities -b> food type, price range, area, address, phone number, postcode, restaurant name and signature dish.
Note - User utterance is in blue and system response is in green.
Our model also had certain weaknesses:
- The model is not able to generate phone numbers, postcodes and addresses correctly. For phone numbers it just utters a random string. This was expected as the system does not have a back-end database which the model can query to get the correct value.
- The model is not the most robust. While it can deal with minor deviations, completely unrelated utterances like “Show me the nearest gas station” lead to unpredictable results. This was expected as the model is trained on specifically on restaurant data and here too it cannot answer questions like “Does the restaurant serve ice-cream?”. Such samples are not present in the training dataset; however this information can be easily implemented using a few utterances with a has_dish intent if required in the future.
- The GPT model cannot support conversations longer than 512 tokens (words/subwords). This is not a big issue, as conversations are rarely that long. The 512 token limit is GPT’s inbuilt structure which cannot be bypassed.
There are a number of ways in which we can improve our model in the future. Future research in this direction should also be done keeping the following points in mind.
- Delexicalization - This means to remove actual examples of restaurant names, addresses, phone numbers, etc. with tokens of the form <restaurant_name>, <address>, <phone_number>, and so on. This can be done wither in the training dataset directly (preferred) or in the system response generated by the conversational system.
- Once delexicalization has been done, we can connect the model to a back-end database using API calls. This will make the model a truly end-to-end conversational system.
- A final way of improving the system is by using GPT-2 instead of GPT. GPT-2 has 1.5 B parameters, which is 10 times more than GPT. Using GPT-2 would lead to better performance, as has been shown for most other NLP tasks due to the larger number of parameters.
In this research project we were able to develop a restaurant information conversational system from scratch. Extensive knowledge has been gained about the working of state-of-the-art NLU and NLG models. The completion criteria that was decided at the start of the project has also been successfully met. We hope that this project can inform and guide future research in hope of simultaneously improving and simplifying the design of task-oriented conversational systems.