Write stopword list for Middle English

[kylepjohnson]

Or, even better, take one from somewhere else.

This could be statistical (some kind of word frequency measures) or grammatical (part of speech types).

Open to anyone.

[tasnani]

I'd like to take this issue. I've begun writing some code to do this. I have some questions though.

1. Would the statistical analysis take place for the corpus collectively or individual files i.e how many times of each word is present in each file or how many words(of same spelling) occurs in all the files collectively?
2. Would frequency measures involve percentage too i.e % of most common words among all stop words? The word count would be essential.
3. Is there a specific number/minimum amount of stop words to be considered? I know considering some stop words in modern english that remains the same with Middle English can increase the stop word list.

[kylepjohnson]

Would the statistical analysis take place for the corpus collectively or individual files

For the corpus

Would frequency measures involve percentage too i.e % of most common words among all stop words?

Simple word frequency is fine.

Is there a specific number/minimum amount of stop words to be considered?

Let's look at the distribution and then talk about it

considering some stop words in modern english that remains the same with Middle English can increase the stop word list

Yes, let's consider Modern English words, too, that also occur in Middle.

One final thing to impress upon you is that our stopword list ought to be based on part-of-speech, not frequency. Frequency is just a good way to find those words. For example, "king" might be a common ME word, however it is not a stopword. Likewise, "ye" might be uncommon, yet it is a stopword.

Also, please report back here what you find from researching this. If we can use a proven ME stopword list that has been composed by expert scholars, then that is probably preferable to what you and I can come up with.

[tasnani]

I found an incomplete list of ME stop words here: https://people.stanford.edu/widner/content/text-mining-middle-ages

[kylepjohnson]

I would feel better about using this list than making our own. I propose the following:

1. You copy these stopwords and add them to the CLTK.
2. I contact the author of this post and ask if he can help check or improve upon what you have done.

How does this sound?

Take a look at the Latin stopwords in the core to see how to add them for middle_english. Write here if you get stuck!

[tasnani]

The list of Modern English words to combine with ME came from here initially: http://www.textfixer.com/resources/common-english-words.txt

[kylepjohnson]

Make a Middle English stopword list. This is easy and open to anyone :)

[michaalbert]

I`d like to do this task if possible. :)

[kylepjohnson]

Yes please!

Do you have any questions about it? Remember that for us, we only want low-information words based on grammar (so articles, pronouns, demonstratives, the verb "to be"). This means we should not include nouns, verbs, adjectives, etc..

I recommend you simply start with transcribing this list, linked to above: https://people.stanford.edu/widner/sites/default/files/Slide13.jpg

How many days would you like for this task?

[michaalbert]

Transcribing the above lists is easy enough of course, however I may have minor questions when writing the test for the stops like in test_stop.py, which probably could be resolved quickly via Gitter. :)

I think I'll be done in a day or two.

[kylepjohnson]

Tests for stops are not necessary, as there is no code associated with them. (I might consider written a simple function for a uniform API to all languages, but list comprehensions are so easy.)