[COMPS 2016-2017] This project is an attempt at developing a conversation machine for helping students find and register for courses at Carleton College. We combined our domain knowledge about Carleton, principles of conversation such as Grice's Maxims of Conversation, and fundamental computer science principles to create a conversational structure and reactive system that can incorporate context and objectives to register a client for a personalized course load.
Visit the COMPS webpage to see a detailed overview of the project: http://cs.carleton.edu/cs_comps/1617/dialogue/final-results/index.html
To set up for your own project, you will need to add the various information to our config.json file. The information that you need to fill out is:
luisurl: The link to your published Microsoft LUIS account. You will get a link when you publish it.
database_host: Wherever you host the database. We set up our database with the help of Mike Tie.
database_name: Whatever the name of your database is.
database_password: The password to your database.
After all of this information is filled out, you will be able to run Making Your Advisor Obsolete.
This project is composed of three organizational components: NLUU: Natural Language Understanding Unit Dialogue Manager Task Manager
The NLUU is responsible for all things relating to natural language in both input and output. We rely on third-party libraries like Python's nltk and Microsoft's LUIS (Language Understanding Intelligent Service) that deal with raw text and break it down or otherwise analyze it in a number of useful ways. The NLUU is also responsible for creating the response to any input that we get from a client. This is accomplished by using a number of templates to build the structure of our responses and then filling in any necessary information with information returned from the Dialogue Manager.
We use nltk’s part of speech tagger to extract parts of sentences that we believe are likely keywords and the VADER sentiment analyzer to identify the sentiment of a client’s utterance, stripping away semantics. Additionally, we use nltk's word stemmer and the operating system's list of words to create a dictionary whose keys are word stems and values are lists of words whose stem is the key. This allows us to effectively expand keywords into a number of related words when querying the database to hopefully provide better results. nltk can be found at http://www.nltk.org/.
We use Microsoft's LUIS to extract intent from a client's response. We have trained the LUIS program on a number of utterances and flagged each as a particular intent. For instance, "Register me for math of cs" yields a ScheduleClass intent, meaning that the client wants to add a class to their schedule. We then use this knowledge of what the client wants to do to determine our next actions. Microsoft LUIS is part of the Azure program, and you can make an Azure account at https://azure.microsoft.com/en-us/, and LUIS can be found at https://www.luis.ai/. We have included our LUIS JSON file so that anyone wishing to extend or use our project can train on top of our existing intents and entities instead of starting over. For our project to run, you will likely need to change the luisurl variable in the config.json file to match the url of the LUIS project that you create. You can read more about LUIS documentation here: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/sigdial2015demo.pdf.
This is the real brains of the program in terms of conversation, decision making and handling. A number of files in this folder are domain objects to help us group piece of information, such as courses and students. Every time the student asks about particular classes, each is filled out and manipulated using a Course object. Each conversation is given a Student object that holds information about previous things we have learned about the student as well as courses we've talked about. These are relatively brainless collections of data that do not provide functionality, except for a pretty way of printing to the terminal. The two main files in this domain are Conversation.py and decision_tree.py.
This file starts the conversation, enters a conversational loop, and handles all of the decision making given the current intent and the past context in which a particular utterance is made. For instance, "Put me in CS111" will give back the same ScheduleClass intent as before, and Conversation will determine that CS111 is a course, query the database (via TaskManager) to fill out the information, and add it to the student's schedule. More interestingly, we handle more complicated interactions that last longer than a single input/output. For instance, say the client is interested in rocks. We query the database about classes that may have something to do with rocks and suggest 3-5 courses for the client to consider. They may wish to register for one of those courses, but colloquially referring to a listed course by its title may sound irregular. One would instead say "Register me for the 2nd one". Being able to understand the intent and the context is integral to producing the correct behavior. This is what Conversation attempts to do.
The decision tree is a tree of nodes that guide the conversation. Each node is linked to a single userQuery object, which correspond to one of nearly forty responses we might give to a user. We ask leading questions, respond to deviations in the trajectory in the conversation made by the client in intelligent ways that still allow us to accomplish our objective of registering the client for 3 courses. The order in which we ask questions and determining how questions have been answered or deflected is handled here. We knew that our eventual goal was to get the student to register for a course, and the tree is built around that fact. We have constructed a series of connections for each node that are dependent on whether or not the user answered the question. This allows a user to control the flow of the conversation if they so choose. We check what information we already have saved in order to ensure we don’t ask a question we already know the answer to. We also try to avoid asking questions we have already asked, unless it is a question worth repeating. By checking for both of these things, our tree allows us to push the conversation in a sensical and practical way that furthers our goal of recommending courses.
The Task Manager acts as an intelligent connection to the database that organizes particular results it returns before sending them back to the Dialogue Manager. It has access to the PostgreSQL server that houses information on courses, professors and a calculated keyword occurrence matrix. The Task Manager uses a wide range of query functions to construct a SQL query, executes that query, and then, depending on the type of request from the Dialogue Manager, decides what courses need to returned first.
The functions that rely on this organization are functions like query_by_keywords and query_by_distribution, where the order of the returned courses has the ability to define what courses the user will look at in a possibly enormous selection of results. The construction of these course objects happens in a helper function called fill_out_courses, which uses the returned SQL query results, and simply fills out the corresponding information in the course object. The actual organization of these courses relies on another helper function called calculate_course_relevance, which assigns a score to the weighted_sum attribute based on the diversity of keywords that occur in the title and description, the total count of keywords that occur, the department the course exists in, and the other interests that the student has previously mentioned, passed to us as student_interests. The function query_by_distribution doesn’t have a list of keywords as input, so instead, the Dialogue Manager passes previous interests, and those are used as keywords to adjust the order of the courses returned.
The weights accompanied to these measures of relevance is arbitrary, chosen by the comps group on the basis that it was an approximate way of representing which courses were more important to students. Weighing options differently could affect what kind of courses appear first in the list of returned courses. Adjusting the weights assigned is as easy as changing the contents of weights, a list of the weights to be assigned in calculate_course_relevance.
The task manager is also responsible for correctly assigning what majors and concentrations students are using based on dictionaries called major_dict and concentration_dict. Functions like major_match and concentration_match use edit distance to determine what major or concentration the student was discussing. The long form is returned for printing purposes (as opposed to the short form four letter code, e.g. “ECON”). Occurrence Matrix The occurrence matrix consists of the series of words in the course descriptions and titles organized in rows, aligned with the occurrence of that particular word in the department that is represented by the column. This is referred to as a text-document matrix in Jurafsky and Martin’s Speech and Language Processing.
In this particular case, we used departments as documents because the individual courses do not contain enough words for the occurrence matrix to be of any use. Jurafsky and Martin recommend 10,000 words per document, hence using the entire department, which should provide enough words to appropriately match the needs of the text-document literature.
The function responsible for creating the occurrence matrix is called make_occurrence_matrix in the Task Manager.
Our project’s components are largely inspired from several informative texts that broke down the various components of a dialogue system, and deconstructed processes that our group felt was integral to the success of the project. Jurafsky and Martin’s Speech and Language Processing was especially foundational in the genesis of our project, allowing us to break down our project into the three main components. The NLUU, Dialogue Manager, and the Task Manager are all components that emerged from this text. This book served to illuminate what role each of these components should have, and gave us a general system architecture that also implied the chronology of processes that would take place for each utterance. The first process for many of these utterances was relation extraction. This extraction was inspired from Zeng et al’s paper “Relation Classification via Convolutional Deep Neural Network”. This idea that relation extraction would be reliant on machine learning was crucial to developing any extraction whatsoever. Initially, we attempted to do relation extraction with NLTK, but the grammars provided with this tool were inadequate in parsing out the entities we needed to perform a variety of queries. Once we established machine learning would be necessary, we used the process referenced to in “Fast and easy language understanding for dialog systems with Microsoft Language Understanding Intelligent Service (LUIS),” and created our own LUIS profile. Once we had trained LUIS on a series of sentences that we felt was representative of our use cases, our product was able to appropriately extract entities and intents. There were several cases where extracting entity or intent isn’t useful in making conversational decisions. For these cases, we were more interested in the sentiment of a user, and making decisions based on these sentiments. We understood this as a possibility after reading “Sentiment Analyzer: Extracting Sentiments about a Given Topic using Natural Language Processing Techniques.” This paper provided us with a framework for approaching this problem. We then appropriated NTLK’s VADER to extract sentiment in a similar manner to the tactic described in the paper.
This project has a lot of potential for growth, both in terms of expanded functionality and overall polish/consistency. While the project managed to integrate most of our initially proposed features, there were a few features which ended up scrapped or only partially implemented due to time constraints. One feature which was considered early in the project’s conception was course time blocks and conflict management. While our final version displays meeting dates/times when available, the system does not account for scheduling conflicts when registering a user for courses. This was a result of the formatting of the meeting times in our data source, and with some additional development time, the conversation could check for time conflicts when registering. Another feature which was partially integrated in the dialogue system is course professors. While our conversation displays the professor of a course when displaying, the system could also consider user preference of instructor, and even query on a user’s favorite professors. Additionally, the quality of results returned from queries in our dialogue system could be improved by the use of Term Frequency/Inverse Document Frequency, or TF/IDF, instead of the simpler occurrence matrix which is currently in use. Currently, our occurrence matrix doesn’t account for the lack of utility that some words have, e.g. “explore” is a word that is not particular to any single department. In the case of “explore”, the departments selected based on this keyword would skew towards departments with more classes. TF/IDF would allow us to account for this, because the Inverse Document Frequency component would factor in the presence of the word in all the departments. You can read more about TF/IDF from the Juan Ramos paper here: https://www.semanticscholar.org/paper/Using-TF-IDF-to-Determine-Word-Relevance-in-Ramos/b3bf6373ff41a115197cb5b30e57830c16130c2c While the conversation system is stable and relatively consistent in its current state, there are a number of ways to improve the overall consistency and versatility. Namely, the dialogue system could benefit from an expanded decision tree which would result in a more natural flow of conversation, with additional branches and userQueries to reflect the wide variety of intents which could potentially appear in a conversation of this domain. Lastly, the LUIS AI component of the system could also benefit from additional intents, more training and improved entity extraction. Perhaps the system could even be built on an alternative to LUIS which is specialized to identify and classify intents/entities unique to the course registration domain of conversation.