Lemma exception tables not reflected in lookup or rule-based lemmatizers

[tarskiandhutch]

Issues related to the implementation of lemmatizer exceptions have appeared on the spaCy Github since at least 2016. From what I can tell, none of them address explicitly the fact that even exceptions from the default lookup tables (or their predecessor data structures) do not seem to be consistently applied by a given lemmatizer. If a similar question has been addressed before, then perhaps the new lemmatizer-as-pipe arrangement merits a renewed explanation.

Is the below expected behavior? If so, then how does one ensure that a given lemmatizer uses the exceptions specified in the lemma_exc lookup table?

How to reproduce the behaviour

After extracting the lemmatizer pipe, we can check in the associated lookup tables that wunderkinder (noun) is keyed to the lemma wunderkind and forbore (verb) to the lemma forbear. But even when the POS is tagged correctly, they are not lemmatized by the pipeline.

>>> import spacy
>>> nlp = spacy.load("en_core_web_lg")
>>> lemmatizer = nlp.get_pipe("lemmatizer")
>>> lemmatizer.lookups.get_table("lemma_exc")["noun"]["wunderkinder"]
['wunderkind']
>>> lemmatizer.lookups.get_table("lemma_exc")["verb"]["forbore"]
['forbear']
>>> doc_1 = nlp("I've known several wunderkinder in my life.")
>>> doc_2 = nlp("He never forbore a smile.")
>>> (doc_1[4].lemma_, doc_1[4].pos_)
('wunderkinder', 'NOUN')
>>> (doc_2[2].lemma_, doc_2[2].pos_)
('forbore', 'VERB')

Furthermore, neither the rule-based nor the lookup-based lemmatizer takes the the table entries for wunderkinder and forbore into account.

>>> lemmatizer.lookup_lemmatize(doc_1[4])
['wunderkinder']
>>> lemmatizer.lookup_lemmatize(doc_2[2])
['forbore']
>>> lemmatizer.rule_lemmatize(doc_1[4])
['wunderkinder']
>>> lemmatizer.rule_lemmatize(doc_2[2])
['forbore']

The above examples are just for illustration purposes. I am asking this question because, in order to rig up a custom lemmatization pipe for my applications, I would like to understand how and when lemma exceptions are applied.

Your Environment

• spaCy version: 3.0.1
• Platform: macOS-10.15.7-x86_64-i386-64bit
• Python version: 3.8.0
• Pipelines: en_core_web_lg (3.0.0), en_core_web_sm (3.0.0)

[adrianeboyd]

The built-in lookup lemmatizer only uses the form-based lookup table. It's intended for cases without POS, so it doesn't use any POS-based exceptions. You can design a custom lemmatizer that uses lookup tables differently, though, see the Polish lemmatizer for an example with POS-based lookup tables.

What you're running into with the rule-based lemmatizer for wunderkinder and forbore is tagging errors + the is_base_form check that looks at POS + morphological features and shortcuts additional processing if it looks like the form should already be the lemma.

The incorrect tags lead to incorrect morphological features (from the tag map patterns in attribute_ruler), and then the is_base_form check mistakenly thinks they don't need to be processed further:

• wunderkinder has NN --> NOUN / Number=Sing (would need to be NNS to be treated as Number=Plur)
• forbore has VBP --> VERB / Tense=Pres|VerbForm=Fin (would need to be VBD to be Tense=Past)

So this behavior isn't a bug exactly, but you might want to have a custom lemmatizer that's less sensitive to these kind of tag details to be more robust around common tagging errors.

You may want to add a new lemmatizer mode with your own custom behavior or write a separate custom lemmatizer? If you're using the English tables from spacy-lookups-data, be aware of the problem described #435, which is part of why there's an is_base_form check for singular nouns. If you have your own custom lemmatizer tables, you may be able to handle exceptions differently.

[tarskiandhutch]

@adrianeboyd Thank you for the rapid and informative response! I had assumed that the rule-based lemmatizer was simpler than it actually is. It's understandable that the problem with these rare inflections is a failure to determine their fine-grained tag. I still have the interpreter open from yesterday, and indeed, as you note (with the verb tag different from yours but still not past tense):

>>> (doc_1[4].lemma_, doc_1[4].pos_, doc_1[4].tag_)
('wunderkinder', 'NOUN', 'NN')
>>> (doc_2[2].lemma_, doc_2[2].pos_, doc_2[2].tag_)
('forbore', 'VERB', 'VB')

I tried to do another test by altering the tables and re-adding them:

>>> import spacy
>>> nlp = spacy.load("en_core_web_lg")
>>> lemmatizer = nlp.get_pipe("lemmatizer")
>>> lemma_index = lemmatizer.lookups.get_table("lemma_index")
>>> lemma_exc = lemmatizer.lookups.get_table("lemma_exc")
>>> lemma_index["verb"].append("schlink")
>>> lemma_exc["verb"]["schlank"] = ["schlink"]
>>> lemma_exc["verb"]["schlunk"] = ["schlink"]
>>> lemmatizer.lookups.remove_table("lemma_index")
>>> lemmatizer.lookups.remove_table("lemma_exc")
>>> lemmatizer.lookups.add_table("lemma_index", lemma_index)
>>> lemmatizer.lookups.add_table("lemma_exc", lemma_exc)

But then I realized that it's (as far as I can tell) impossible to get the tagger to guess that the nonce word (i.e., schlank, schlunk) is an inflected form – it always errs in favor of a base form, or sometimes a different POS altogether.

>>> doc_3 = nlp("He had not schlunk in years.")
>>> doc_3[3].lemma_, doc_3[3].tag_
('schlunk', 'VB')
>>> doc_4 = nlp("Three years ago, I schlank for the very first time.")
>>> doc_4[5].lemma_, doc_4[5].tag_
('schlank', 'VBP')
>>> doc_5 = nlp("Have you ever schlunk before?")
>>> doc_5[3].lemma_, doc_5[3].tag_
('schlunk', 'VB')
>>> doc_6 = nlp("He had schlunk across the room with great effort.")
>>> doc_6[2].lemma_, doc_6[2].tag_
('schlunk', 'NN')

It would seem that rule-based lemmatizer exceptions are most of the time a catch-22, no? If an inflection is so rare as to require a lemmatizer exception, then it is likely also too rare to have appeared with enough frequency in the tagger's training data for the tagger to recognize it later on as a conjugated verb, pluralized noun, etc. And so the best option is, like you suggest, to relax the rules of the lemmatizer.

Thank you again!

[github-actions]

This issue has been automatically closed because it was answered and there was no follow-up discussion.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.