README.md

@@ -26,6 +26,12 @@ the quality of answers but it makes the numerical results non-reproducible
 as the web search results can change day to day.  Please use the "master"
 branch for day-to-day development.
 
+**Bing search is disabled by default. You need to set system property
+  cz.brmlab.yodaqa.use_bing=yes to enabled it. 
+  You can run yodaqa with this command:**
+
+  ./gradlew run -q -Dcz.brmlab.yodaqa.use_bing=yes  
+
 ## Installation Instructions
 
 Quick instructions for setting up, building and running (focused on Debian Wheezy):
@@ -51,20 +57,16 @@ frontend which offers a prompt and answers questions interactively;
 answer candidates and their confidence score are listed after a while
 (the first question takes a bit longer to answer as the models etc. are
 loaded).
-
 Alternatively, you can use the "web" frontend by executing
 ``./gradlew web -q`` and opening e.g. http://localhost:4567/ in your browser.
-It is also possible to let YodaQA answer many questions at once, e.g. to
-measure the performance; use ``./gradlew tsvgs`` to feed YodaQA
-the curated testing dataset from data/eval/.  (See also data/eval/README.md
-for more details, and a convenient wrapper script ``train-and-eval.sh``.)
-To connect YodaQA to IRC, see ``contrib/irssi-brmson-pipe.pl``.
+A shinier web interface is available at https://github.com/brmson/YodaQA-client
+and you can also use the web frontend as a REST API.
 
 By default, there is a lot of output regarding progress of the answering
 process; redirect stderr, e.g. ``2>/dev/null``, to get rid of that.
-Alternatively, if things don't go well, try passing an extra parameter
-``-Dorg.slf4j.simpleLogger.defaultLogLevel=debug`` on the commandline,
-or specifically ``-Dorg.slf4j.simpleLogger.log.cz.brmlab.yodaqa=debug``.
+Alternatively, if things don't go well or you would like to watch YodaQA
+think, try passing an extra command line parameter
+``-Dorg.slf4j.simpleLogger.log.cz.brmlab.yodaqa=debug`` to gradle.
 
 Sometimes, Java may find itself short on memory; don't try to run YodaQA
 on systems with less than 8GB RAM.  You may also need to tweak the
@@ -73,6 +75,12 @@ on a 32-bit system.  By default, YodaQA will try to use *half* of the logical
 CPU cores available; set the YODAQA_N_THREADS environment variable to change
 the number of threads used.
 
+It is also possible to let YodaQA answer many questions at once, e.g. to
+measure the performance; use ``./gradlew tsvgs`` to feed YodaQA
+the curated testing dataset from data/eval/.  (See also data/eval/README.md
+for more details, and a convenient wrapper script ``train-and-eval.sh``.)
+To connect YodaQA to IRC, see ``contrib/irssi-brmson-pipe.pl``.
+
 ## Data Sources
 
 YodaQA uses Solr fulltext indexing framework as a data source, either
@@ -110,20 +118,17 @@ to page id we can use to fetch the page from solr) and redirect walking.
 By default, we rely on a DBpedia-2014 SPARQL endpoint running on the author's
 computer.  In case it is offline, you can try to switch it to the public
 DBpedia SPARQL endpoint, though it is prone to outages and we shouldn't use
-it too heavily anyway.  Simply edit the ``service`` attribute value in file
-``src/main/java/cz/brmlab/yodaqa/provider/rdf/CachedJenaLookup.java``.
-
-Detailed instrutions for setup of local DBpedia SPARQL endpoint can be found
-in ``data/dbpedia/README.md``.
+it too heavily anyway, or you can fairly easily set up a local instance of
+DBpedia.  Detailed instrutions can be found in ``data/dbpedia/README.md``.
 
 ### Freebase Data Source
 
 We can also leverage another structured data source, the Freebase.
 We use its RDF export with SPARQL endpoint, running on infrastructure
 provided by the author's academic group (Jan Šedivý's 3C Group at the
 Dept. of Cybernetics, FEE CTU Prague).  If the endpoint is not available
-for some reason, you can also disable Freebase usage by commenting
-out the fbo.query() line in the code of:
+for some reason, you can also disable Freebase usage by editing the
+method getConceptProperties() (instructions inside) of:
 
 	src/main/java/cz/brmlab/yodaqa/pipeline/structured/FreebaseOntologyPrimarySearch.java
 

build.gradle

@@ -49,6 +49,18 @@ task(tsvgs, dependsOn: 'classes', type: JavaExec) {
   systemProperties = System.getProperties()
 }
 
+task(jsongs, dependsOn: 'classes', type: JavaExec) {
+  main = 'cz.brmlab.yodaqa.YodaQA_JsonGS'
+  if (project.hasProperty('execArgs'))
+    args(execArgs.split(' '))
+  else
+    args = ['data/eval/output_train.json', 'json_results.tsv']
+  minHeapSize = "2048m"
+  maxHeapSize = "4500m"
+  classpath = sourceSets.main.runtimeClasspath
+  systemProperties = System.getProperties()
+}
+
 task(questionDump, dependsOn: 'classes', type: JavaExec) {
   main = 'cz.brmlab.yodaqa.QuestionDump'
   if (project.hasProperty('execArgs'))

data/bing/README.md

@@ -61,6 +61,14 @@ Once you key the key int he file, the application starts using snippets
 from bing search results as an answer source. If the file is not presented or it does not contain proper API key,
 the application skips bing search.
 
+Enable bing
+-----------
+
+Bing search is disabled by default. You need to set system property "cz.brmlab.yodaqa.use_bing=yes" to enabled it.
+You can run yodaqa with this command:
+
+	./gradlew run -q -Dcz.brmlab.yodaqa.use_bing=yes	
+
 Caching results
 ---------------
 

data/eval/README.md

@@ -80,6 +80,12 @@ rid of old data, use:
 
 (WARNING: AUTOMATIC FILE REMOVAL, NO WARRANTY.)
 
+There is also a newer alternative way to batch-query a running YodaQA
+for a set of questions and evaluate the replies, using the REST API and
+questions stored in JSON files:
+
+	data/eval/rest-eval.py ../dataset-factoid-webquestions/main/devtest.json http://localhost:4567/
+
 Legacy Benchmarking
 -------------------
 

data/eval/_multistage_traineval.sh

@@ -30,6 +30,7 @@ dataset="$2"
 retrain="$3"
 wait_for_train="$4"
 basecommit="$5"
+system_property="$6"
 args0=
 argsF=
 
@@ -101,6 +102,7 @@ if [ -z "$basecommit" ]; then
 		-PexecArgs="$basedir/data/eval/${dataset}.tsv $outfile0" \
 		-Dorg.slf4j.simpleLogger.log.cz.brmlab.yodaqa=debug \
 		-Dcz.brmlab.yodaqa.save_answerfvs="$xmidir" \
+		$system_property \
 		$args0
 	base_xmidir="$xmidir"
 	base_atrainfile0="$atrainfile0"
@@ -120,6 +122,7 @@ time ./gradlew tsvgs \
 	-Dorg.slf4j.simpleLogger.log.cz.brmlab.yodaqa=debug \
 	-Dcz.brmlab.yodaqa.load_answerfvs="$base_xmidir" \
 	-Dcz.brmlab.yodaqa.save_answerfvs="$xmidir" \
+	$system_property \
 	$args0
 
 
@@ -128,6 +131,7 @@ time ./gradlew tsvgs \
 	-Dorg.slf4j.simpleLogger.log.cz.brmlab.yodaqa=debug \
 	-Dcz.brmlab.yodaqa.load_answerfvs="$xmidir" \
 	-Dcz.brmlab.yodaqa.save_answer1fvs="$xmidir"1 \
+	$system_property \
 	$args1
 
 train_and_sync "1" "$atrainfile1" "$modelfile1"
@@ -138,6 +142,7 @@ time ./gradlew tsvgs \
 	-Dorg.slf4j.simpleLogger.log.cz.brmlab.yodaqa=debug \
 	-Dcz.brmlab.yodaqa.load_answer1fvs="$xmidir"1 \
 	-Dcz.brmlab.yodaqa.save_answer1fvs="$xmidir"1 \
+	$system_property \
 	$args1
 
 
@@ -146,6 +151,7 @@ time ./gradlew tsvgs \
 	-Dorg.slf4j.simpleLogger.log.cz.brmlab.yodaqa=debug \
 	-Dcz.brmlab.yodaqa.load_answer1fvs="$xmidir"1 \
 	-Dcz.brmlab.yodaqa.save_answer2fvs="$xmidir"2 \
+	$system_property \
 	$args2
 
 train_and_sync "2" "$atrainfile2" "$modelfile2"
@@ -156,6 +162,7 @@ time ./gradlew tsvgs \
 	-Dorg.slf4j.simpleLogger.log.cz.brmlab.yodaqa=debug \
 	-Dcz.brmlab.yodaqa.load_answer2fvs="$xmidir"2 \
 	-Dcz.brmlab.yodaqa.save_answer2fvs="$xmidir"2 \
+	$system_property \
 	$args2
 
 

data/eval/rest-eval.py

@@ -0,0 +1,67 @@
+#!/usr/bin/python
+#
+# Script that sends questions from JSON file to the YodaQA web frontend
+# REST API and prints results for error evaluation+analysis.
+#
+# Argument 1: JSON filename
+# Argument 2: YodaQA URL (either "http://localhost:4567" or "http://qa.ailao.eu:4000")
+
+import requests
+import json
+import sys
+from time import sleep
+
+def byteify(input):
+    if isinstance(input, dict):
+        return {byteify(key):byteify(value) for key,value in input.iteritems()}
+    elif isinstance(input, list):
+        return [byteify(element) for element in input]
+    elif isinstance(input, unicode):
+        return input.encode('utf-8')
+    else:
+        return input
+
+argv = sys.argv
+filename = argv[1]
+URL = argv[2]
+json_data = open(filename)
+parsed_data = byteify(json.load(json_data))
+number_of_questions = len(parsed_data)
+question_counter = 0
+correctly_answered = 0
+recall = 0
+finished = False
+
+print('%04s\t%.50s\t%.10s\t%.15s\t%.15s\t%s' % ("ID", "Question Text".ljust(50), "indicator", "correct answer".ljust(15), "found".ljust(15), "URL"))
+
+while question_counter < number_of_questions:
+    questionText = parsed_data[question_counter]["qText"]
+    questionAnswer = parsed_data[question_counter]["answers"][0]
+    ID = parsed_data[question_counter]["qID"]
+    finished = False
+    indicator = "incorrect"
+    r = requests.post(URL+"/q", data={'text':questionText} )
+    current_qID = byteify(r.json()["id"])
+    while (finished == False): #wait for web interface to finish
+        sleep(0.5)
+        data = requests.get(URL +"/q/"+ current_qID).json()
+        finished = data["finished"]
+
+    answer_list = byteify(data["answers"])
+    for i in range (0, len(answer_list)): #iterate through answers and look for our correct one
+        if (questionAnswer == answer_list[i]['text']): 
+            if (i == 0):
+                correctly_answered += 1
+                indicator = "correct  "
+                continue
+            else:
+                recall	 += 1
+                indicator = "recall   "
+                continue
+    print('%03s\t%.50s\t%.10s\t%.15s\t%.15s\t%s' % (ID, questionText.ljust(50),indicator, questionAnswer.ljust(15), answer_list[0]['text'].ljust(15), (URL+"/q/"+str(current_qID))))
+    question_counter += 1
+
+print("correctly answered: " + str(correctly_answered))
+print("recall: " + str(recall))
+incorrect = number_of_questions-(recall+correctly_answered)
+print("incorrect: "+ str(incorrect))

data/eval/train-and-eval.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Usage: data/eval/train-and-eval.sh [-s N] [-m MAXHEAPSIZE] [-d DATASET] [COMMIT [BASECOMMIT]]
+# Usage: data/eval/train-and-eval.sh [-s N] [-m MAXHEAPSIZE] [-d DATASET] [-DPROPERY] [COMMIT [BASECOMMIT]]
 #
 # Perform full model training and performance evaluation of the given
 # commit (may be also a branch name, or nothing to eval the HEAD).
@@ -37,6 +37,9 @@
 # -d DATASET allows "train and eval" on a different dataset than
 # "curated".  E.g. -d large2180 will test on the 2180-question
 # noisier dataset.
+# 
+# You can specify system property with -D. At this time there is only one
+# system property supported: -Dcz.brmlab.yodaqa.use_bing=yes
 
 set -e
 
@@ -55,6 +58,10 @@ if [ "$1" = "-d" ]; then
 else
 	dataset=curated
 fi
+if [ "${1:0:2}" = "-D" ]; then
+	system_property=$1; shift
+fi
+
 
 cid=$(git rev-parse --short "${1:-HEAD}")
 baserepo=$(pwd)
@@ -95,9 +102,9 @@ else
 fi
 
 screen -m sh -c "
-	$run_split \"$baserepo\"/data/eval/_multistage_traineval.sh \"$baserepo\" \"${dataset}-train\" 1 0 $basecid;
+	$run_split \"$baserepo\"/data/eval/_multistage_traineval.sh \"$baserepo\" \"${dataset}-train\" 1 0 \"$basecid\" \"$system_property\";
 	if [ $wait_on_barriers = 0 ]; then rm _multistage-barrier*; else sleep 10; fi
-	$run_split \"$baserepo\"/data/eval/_multistage_traineval.sh \"$baserepo\" \"${dataset}-test\" 0 $wait_on_barriers $basecid
+	$run_split \"$baserepo\"/data/eval/_multistage_traineval.sh \"$baserepo\" \"${dataset}-test\" 0 \"$wait_on_barriers\" \"$basecid $system_property\"
 "
 
 popd

src/main/java/cz/brmlab/yodaqa/QuestionDump.java

@@ -3,7 +3,7 @@
 import cz.brmlab.yodaqa.analysis.question.QuestionAnalysisAE;
 import cz.brmlab.yodaqa.flow.MultiCASPipeline;
 import cz.brmlab.yodaqa.flow.asb.ParallelEngineFactory;
-import cz.brmlab.yodaqa.io.collection.CollectionQuestionReader;
+import cz.brmlab.yodaqa.io.collection.TSVQuestionReader;
 import cz.brmlab.yodaqa.io.debug.QuestionPrinter;
 import org.apache.uima.analysis_engine.AnalysisEngineDescription;
 import org.apache.uima.collection.CollectionReaderDescription;
@@ -24,9 +24,9 @@ public static void main(String[] args) throws Exception {
         }
 
         CollectionReaderDescription reader = createReaderDescription(
-                CollectionQuestionReader.class,
-                CollectionQuestionReader.PARAM_TSVFILE, args[0],
-                CollectionQuestionReader.PARAM_LANGUAGE, "en");
+                TSVQuestionReader.class,
+                TSVQuestionReader.PARAM_TSVFILE, args[0],
+                TSVQuestionReader.PARAM_LANGUAGE, "en");
 
         AnalysisEngineDescription pipeline = QuestionAnalysisAE.createEngineDescription();
 

src/main/java/cz/brmlab/yodaqa/YodaQA_GS.java

@@ -1,12 +1,12 @@
 package cz.brmlab.yodaqa;
 
 import org.apache.uima.analysis_engine.AnalysisEngineDescription;
-import org.apache.uima.collection.CollectionReaderDescription;
 
 import cz.brmlab.yodaqa.flow.MultiCASPipeline;
 import cz.brmlab.yodaqa.flow.asb.ParallelEngineFactory;
-import cz.brmlab.yodaqa.io.collection.CollectionQuestionReader;
+import cz.brmlab.yodaqa.io.collection.TSVQuestionReader;
 import cz.brmlab.yodaqa.io.collection.GoldStandardAnswerPrinter;
+import org.apache.uima.collection.CollectionReaderDescription;
 import cz.brmlab.yodaqa.pipeline.YodaQA;
 
 import static org.apache.uima.fit.factory.AnalysisEngineFactory.createEngineDescription;
@@ -25,9 +25,9 @@ public static void main(String[] args) throws Exception {
 		}
 
 		CollectionReaderDescription reader = createReaderDescription(
-				CollectionQuestionReader.class,
-				CollectionQuestionReader.PARAM_TSVFILE, args[0],
-				CollectionQuestionReader.PARAM_LANGUAGE, "en");
+				TSVQuestionReader.class,
+				TSVQuestionReader.PARAM_TSVFILE, args[0],
+				TSVQuestionReader.PARAM_LANGUAGE, "en");
 
 		AnalysisEngineDescription pipeline = YodaQA.createEngineDescription();
 

src/main/java/cz/brmlab/yodaqa/YodaQA_JsonGS.java

@@ -0,0 +1,46 @@
+package cz.brmlab.yodaqa;
+
+import cz.brmlab.yodaqa.io.collection.JSONQuestionReader;
+import org.apache.uima.analysis_engine.AnalysisEngineDescription;
+import org.apache.uima.collection.CollectionReaderDescription;
+
+import cz.brmlab.yodaqa.flow.MultiCASPipeline;
+import cz.brmlab.yodaqa.flow.asb.ParallelEngineFactory;
+import cz.brmlab.yodaqa.io.collection.GoldStandardAnswerPrinter;
+import cz.brmlab.yodaqa.pipeline.YodaQA;
+
+import static org.apache.uima.fit.factory.AnalysisEngineFactory.createEngineDescription;
+import static org.apache.uima.fit.factory.CollectionReaderFactory.createReaderDescription;
+
+
+/* FIXME: Massive code duplication of YodaQA_Interactive and YodaQA_GS.
+ * Let's abstract out the processing pipeline later. */
+
+public class YodaQA_JsonGS {
+	public static void main(String[] args) throws Exception {
+		if (args.length != 2) {
+			System.err.println("Usage: YodaQA_GS INPUT.json OUTPUT.TSV");
+			System.err.println("Measures YodaQA performance on some Gold Standard questions.");
+			System.exit(1);
+		}
+
+		CollectionReaderDescription reader = createReaderDescription(
+				JSONQuestionReader.class,
+				JSONQuestionReader.PARAM_JSONFILE, args[0],
+				JSONQuestionReader.PARAM_LANGUAGE, "en");
+
+		AnalysisEngineDescription pipeline = YodaQA.createEngineDescription();
+
+		AnalysisEngineDescription printer = createEngineDescription(
+				GoldStandardAnswerPrinter.class,
+				GoldStandardAnswerPrinter.PARAM_TSVFILE, args[1],
+				ParallelEngineFactory.PARAM_NO_MULTIPROCESSING, 1);
+
+		ParallelEngineFactory.registerFactory(); // comment out for a linear single-thread flow
+		/* XXX: Later, we will want to create an actual flow
+		 * to support scaleout. */
+		MultiCASPipeline.runPipeline(reader,
+				pipeline,
+				printer);
+	}
+}