-
Notifications
You must be signed in to change notification settings - Fork 4
Mean Conflation Class Size
"The mean number of words per conflation class — This is the average number of words that correspond to the same stem for a corpus. For example if the words "engineer," "engineered," and "engineering" are stemmed to "engineer," then this conflation class size is three. Stronger stemmers will tend to have more words per conflation class." Frakes & Fox Ref: W1.
This metric is obviously dependent on the number of words processed, but for a word collection of given size, a higher value indicates a heavier stemmer. The value is easily calculated as follows: MWC = Mean number of words per conflation class N = Number of unique words before Stemming S = Number of unique stems after Stemming
MWC = N/S
Verification
Use test file react.txt as File A and react_trunc5.txt as File B.
Mean Conflation Class size = 9/1 = 9
Compression Factor = (9 - 1)/9 = 0.889 rounded to 3 decimal places.
Mean Characters Removed (0 + 1 + 3 + 2 + 3 + 4 + 3 + 5 + 7) /9 = 3.111
MHD = HD(1,P) + (Q-P) where HD(1,P) is the Hamming Distance for the first P characters of both strings.
Mean MHD = the average MHD value for every word in the original sample = 3.111
Inverse Mean MHD = 1/3.111 = 0.321 rounded to 3 decimal places