-
Notifications
You must be signed in to change notification settings - Fork 4
Errors in Lancaster University Stemming Evaluation Tool
Lancaster University developed a suite of tools in Java for producing lists of words with barriers and for measuring error counts (Under & Over stemming, and Error Rate Relative to Truncation (ERRT)). While developing our C# software to include error counting based on the Lancaster University methods we discovered discrepancies between our results and the results in published papers. We discovered that the differences were caused by a couple of issued with the Java software, and these are detailed below.
In Hooper's original Stemmer Evaluation Tool*, on line 802, when it comes time for calculating the Error Rate Relative to Truncation (ERRT), a class which is used to perform the calculation is constructed in the following way (Note the 2nd and last arguments are both weakOILarray):
errt = new ERRT(weakUIarray, weakOILarray, strongUIarray, weakOILarray);
However, when in ERRT.java the constructor's method takes the following arguments:
public ERRT(double[] wUI, double[] wOIL, double[] sUI, double[] sOIL)
Following the naming convention, wUI is short for weakUI, wOIL for weakOIL, sUI for strongUI, and sOIL for strongOIL; in the Java program, when this method gets called, the weakOILarray is put in the place of the strongOIL in the constructor. There is also evidence to suggest that this was in error - Hooper never fully verified the results calculated were correct in his paper Hooper, R (2004) The Design and Optimisation of Stemming Algorithms (Masters Thesis) Page 23, and the naming convention continues throughout the ERRT class.
Taking these arguments, the SW values given would still look like a reasonable answer, but are the wrong calculation; mixing the Strong and the Weak arrays in the ERRT calculations, the result is that for Strong+Weak Barriers, the values ERRT(L) and SW(L) are likely incorrect.
In our C# corrected version of the algorithm, strongOILarray is passed as the last argument to the ERRT constructor, matching in with the logic inside the ERRT class.
The original, unmodified Java Stemmer Evaluation Tool silently fails to write out all the words, leading to the Porter file missing words towards the end of the document - for example, when testing the Porter stemmer with commonwords.txt the last word group to enter the Porter stemmer is for ‘worktables', however in commonwords.txt the last group starts with a z - 'zoom', this will affect the results.
On examination of the Java source code, this is due to file streams being opened and never closed - the remaining data remains in the computer’s 1024 byte default buffer and is never written out to the file - see http://stackoverflow.com/questions/13426142/bufferedwriter-not-writing-everything-to-its-output-file
Below is a table showing the differences with and without the file bug (With ERRT inputs mismatching as in the original):
| Buffer Bug | No Bug | Buffer Bug | No Bug | ||
|---|---|---|---|---|---|
| UI | 0.3821 | 0.3834 | 0.3019 | 0.3027 | |
| OI(G) | 0.0000099 | 0.0000097 | 0.0000198 | 0.0000193 | |
| OI(L) | 0.0788 | 0.0781 | 0.1568 | 0.1555 | |
| ERRT(L) | 61.66 | 61.61 | 55.08 | 54.99 | |
| SW(L) | 20.61 | 20.36 | 26.10 | 25.78 |
Most changes because of this bug were small, however noteworthy.
List Analyzer was modified to include an option on the File>Preferences menu to have the ERRT inputs mismatching, it was then possible to reproduce the 'no bug' results to the same accuracy.
The ERRT Values returned using the Java Stemmer Evaluation Tool (61.66 for Strong Barriers, and 55.08 for Strong + Weak Barriers) match exactly with the results shown in Hooper's paper, page 41 Word List B. Because the ERRT is the final calculated value and matches, it's believed that the other calculated values must be correct and the program version we use is the same as the one used in Hooper's Paper.
Whilst the List Analyzer results do not match the Java Program results, they do match the Java Program results if the File buffer bug in the Java program is removed.
*Hooper's original paper (Rob Hooper - The Design and Optimization of Stemming Algorithm's - Msc) doesn't appear to be available on the Lancaster University website, however a copy of his report can be found on the web.archive.org site https://web.archive.org/web/20060828230713/http://www.comp.lancs.ac.uk:80/computing/research/stemming/Files/Report.zip