Distributed-Question Answering-NLP

Distributed Question Answering system using Natural language processing. There are three modules involved: Question processing, Passage retrieval and Answer processing. Question processing used NLP techniques like Part-Of-Speech tagging and named entity recognition to identify the type of question(who/when/what) and the type of answer(Place/Person/Thing/Date). Passage retrieval used Haoop map reduce and Pig to extract plain-text from WikiPedia XML dumps and store in Pig table for faster access. Answering processing is a Pig module which takes in the set of documents retrieved by Hadoop Map reduce jobs.Custom UDF's were used to filter passaged which matched a given query.Tf-IDF scores and proximity/distance based measures were used as scores to rank the passages. HBase was used to store the IDF values for the entire dictionary. Used Map reduce technologies like Hadoop/Pig/HBase to retreive knowledge base, query Question, Score and rank possible answer passages.