Skip to content

An exploration of the building blocks of an automated Question Answering system. Submitted as a term project for a course in Natural Language Processing.

Notifications You must be signed in to change notification settings

sepiatone/nlp-question_answering

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

18 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Overview

Question Answering is a subfield in Natural Language Processing dealing with automated systems which respond with a suitable answer to a question from the user.

A project for MIT 6.864 - Advanced Natural Language Processing, Fall 2017 done in collaboration with Raoul Khouri

In general, the task of Question Answering - i.e, providing an answer to a question posed (for example on a site like Stack Exchange is hard to automate. A system for this must parse the natural language query to determine what is being asked, then query its knowledge base for a suitable answer (Information Retrieval) and then construct a response (Natural Language Generation).

In this project we build some of the pieces of an automated system that can handle this task of Question Answering to explore the methods involved.

Task - Question Retrieval

We are given a set of questions Q and a training set of similar question pairs {(q1; q11, q12, ...), (q2; q21, q22, ...), ...}. Given a new question q, we have to return all similar question from Q.

The model will be trained on the AskUbuntu dataset. The algorithm is based on the paper Semi-supervised Question Retrieval with Gated Convolutions

Task - Transfer Learning

Here we work on the Android dataset.

We first apply our model trained on the AskUbuntu dataset on the Android dataset, i.e, direct transfer without domain adaptation. We then apply domain adapatation using techniques explained in Unsupervised Domain Adaptation by Backpropagation and Aspect-augmented Adversarial Networks for Domain Adaptation.

Baseline metrics are obatained by using in-house implementations of the BM25 and TF-IDF algorithms.

Framework from Tao Lei

About

An exploration of the building blocks of an automated Question Answering system. Submitted as a term project for a course in Natural Language Processing.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages