Data & Evaluation

##Data

The date used for the experiments were provided by SemEval and defined as follows:

Participants will be presented with a training and a test set. The training set will be composed of 2,237 instances, and the test set of 88,221. The data was collected through a survey, in which 400 annotators were presented with several sentences and asked to select which ones they did not understand the meaning of. The training set is composed by the judgments of 20 distinct annotators over a set of 200 sentences, while the test set is composed by the judgments made over 9,000 sentences by only one annotator. In the training set, a word is considered to be complex if at least one of the 20 annotators judged them so. [src]

Take a look at the SemEval's readme or click here for more information on the data and its formatting convention.

In this project both datasets were used in cross validation experiments. The smaller training data were used to determine the quality of different features and classifiers quickly, before checking these findings on the annotated test data.

---

##Classification Performance
To evaluate the tools's performance, the measures Accuracy, Precision & Recall and F-Measure were calculated on different configurations. In the following, evaluation results will be summarised. The actual data can be accesssed here.

Generally speaking, classification performance is rather poor.

Performance was better on the smaller dataset. This might be in part due to the fact, that it generally contained a larger portion of complex words (26,37%) compared to the larger dataset (4,68%). Probabalistic classifiers might not be the best choice for this setup.

####Experiment Setup
Both experiments used the same parameter space, to keep them comparable and only differed in the classifier used. The following parameters yielded the best results (parameters were systematically varied to get to this result):

ExperimentType: Cross Validation
Labeling Mode: Single Label
Feature Mode: Unit

Mandatory Preprocessing: Tokenisation, Lemmatisation, POSTagging, (lookup based) Frequency Tagging
Features Extracted: Word Length (in characters), POS, Character NGrams (min:2, max:4)

More details on the algorithms can be found in the Code] Section.
Regarding the number of folds, 10 seemed to yield best results while still maintaining a managable runtime.

#####ZeroR

As mentionied before, the ZeroR Algorithm has proven to be unuseful in complex word identification. This should come at no surprise, as complex words are less frequently used than non-complex ones. Moreover, no insights can be gained as to which factors influence word difficulty.
The results serve as a good example for the potentially misleading nature of classifiers' Accuracy Rating. Despite its acceptable accuracy, ZeroR is entirely unsuited for complex word identification.

Accuracy: 73,60%
Precision: 0%
Recall: 0%
F-Measure: 0,0

#####Naive Bayes

Metrics are slightly better for the Naive Bayes classifier. Of all the classifiers, its recall value was the highest (65,01%). It was the only classifier, to actually overestimate the number of complex words, making it the best choice, when trying to find all complex words in a difficult text. Amongst these, there will be, however an overwhelming majority of non-complex words, due to its low precision (13% on the large corpus).

Small Corpus(n = 2044)
Accuracy: 54,72%
Precision: 37,47%
Recall: 65,01%
F-Measure: 47,54%

Large Corpus (n = 84090)
Accuracy: 86,22%
Precision: 13,00%
Recall: 34,13%
F-Measure: 0,188

(Runtime on large corpus: ~15 Minutes)

#####J48

The J48 Decision Tree produced the best overall results (F-Measure: 0,501 on the small Corpus). Its runtime is, however, significantly higher than that of Naive Bayes. Especially on the Large corpus, this was a problem, as a single could take multiple hours. Its Precision on the large corpus was surprisingly high (43,56%), compared to the other metrics.

Small Corpus(n = 2044)
Accuracy: 54,72%
Precision: 37,47%
Recall: 65,01%
F-Measure: 0,500

Large Corpus (n = 84090)
Accuracy: 94,95%
Precision: 43,56%
Recall: 10,16%
F-Measure: 0,165

(Runtime on large corpus: ~180 Minutes)

#####Random Forest

While having the weakest overall performance on the small dataset (F-Measure: 0,424), the Random Forest Decision Tree Algorithm actually has the best F-Measure on the large Dataset ( F-Measure: 0,213 ). Even though the performance on large corpora is still rather poor, the Random Forest algorithm is the most promising candidate in terms of performance / runtime ratio.

Small Corpus(n = 2044) Accuracy: 68,57%
Precision: 50,29%
Recall: 36,69%
F-Measure: 0,424

Large Corpus (n = 84090)
Accuracy: 94,95%
Precision: 43,56%
Recall: 10,16%
F-Measure: 0,213

(Runtime on large corpus: ~70 Minutes)

---

###Conclusions

As mentioned before, overall performance was extremely poor on the large corpus, which would imply, that no safe conclusions can be drawn, as to whether or not the examined features are actually useful predictors for word difficulty.