[BUG] Apostrophes

[nicno90]

Removing "s" after apostrophe
When apostrophes are in sentence yields weird results.

To Reproduce

#Steps to reproduce the behavior:
1. Init spacy model '...'
2. Add contextualSpellCheck '....'
3. supply the sentence "Spell Checking based on Peter Norvig’s blog post."
4. doc._.outcome_spellCheck gives result: "Spell Checking based on Peter Norvig, blog post."

Expected behavior
" 's " should not be touched.

Version (please complete the following information):

• contextualSpellCheck [e.g. 0.3.0]
• Spacy: [e.g. 2.3.2]
• transformers [e.g. 3.1.0]

[R1j1t]

I tried running the given sentence and it worked as expected on my local. If there is no spelling correction then it returns '' as mentioned in the README.

>>> import spacy
>>> import contextualSpellCheck
>>> nlp = spacy.load("en") 
>>> contextualSpellCheck.add_to_pipe(nlp)
<spacy.lang.en.English object at 0x7f911cef4a90>
>>> nlp.pipe_names
['tagger', 'parser', 'ner', 'contextual spellchecker']
>>> doc = nlp('Spell Checking based on Peter Norvig’s blog post.')
>>> doc._.outcome_spellCheck
''
>>> 

Please provide the code to replicate and version info of the package if you are not using the latest.

[nicno90]

nlp = spacy.load('en_core_web_lg')
spell_checker = ContextualSpellCheck(max_edit_dist=4)
nlp.add_pipe(spell_checker)


def correct_spelling(sentence):
    global nlp
    doc = nlp(sentence)
    return doc._.outcome_spellCheck


print(correct_spelling("Pure Python Spell Checking based on Peter Norvig’s blog post on setting up a simple spell checking algorithm."))

Version:

• contextualSpellCheck == 0.3.0
• Spacy == 2.3.2
• transformers == 3.3.1

[R1j1t]

Thanks @nicno90 I will have a look.

[R1j1t]

I checked and have found the issue.

When using en_core_web_sm, token 's is identified as PERSON from spaCy pipeline but in en_core_web_lg it is not identified as PERSON. Below is my logic to identify misspell.

contextualSpellCheck/contextualSpellCheck/contextualSpellCheck.py

Lines 242 to 254 in 96cb79e

	if (
	(token.text.lower() not in self.vocab)
	and (token.ent_type_ != "PERSON")
	and (not token.like_num)
	and (not token.like_email)
	and (not token.like_url)
	# added after 0.0.4
	and (not token.is_space)
	and (not token.is_punct)
	and (token.ent_type_ != "GPE")
	and (token.ent_type_ != "ORG")
	):
	misspell.append(token)

token.ent_type_ != "PERSON" is causing the issue. I will try to fix it and release it by next weekend.

Thank you for identifying this issue!

[nicno90]

@R1j1t Thank you for looking into it!

[R1j1t]

@nicno90 I have added the fix and test cases. I have released it to PyPi in v0.3.2