Multi Word Expressions (v0.3)

[apmoore1]

Added

• Roadmap added.
• Define the MWE template and it's syntax, this is stated in Notes -> Multi Word Expression Syntax in the Usage section of the documentation. This is the first task of issue #24.
• PEP 561 (Distributing and Packaging Type Information) compatible by adding py.typed file.
• Added srsly as a pip requirement, we use srsly to serialise components to bytes, for example the pymusas.lexicon_collection.LexiconCollection.to_bytes function uses srsly to serialise the LexiconCollection to bytes.
• An abstract class, pymusas.base.Serialise, that requires sub-classes to create two methods to_bytes and from_bytes so that the class can be serialised.
• pymusas.lexicon_collection.LexiconCollection has three new methods to_bytes, from_bytes, and __eq__. This allows the collection to be serialised and to be compared to other collections.
• A Lexicon Collection class for Multi Word Expression (MWE), pymusas.lexicon_collection.MWELexiconCollection, which allows a user to easily create and / or load in from a TSV file a MWE lexicon, like the MWE lexicons from the Multilingual USAS repository. In addition it contains the functionality to match a MWE template to templates stored in the MWELexiconCollection class following the MWE special syntax rules, this is all done through the mwe_match method. It also supports Part Of Speech mapping so that you can map from the lexicon's POS tagset to the tagset of your choice, in both a one-to-one and one-to-many mapping. Like the pymusas.lexicon_collection.LexiconCollection it contains to_bytes, from_bytes, and __eq__ methods for serialisation and comparisons.
• The rule based taggers have now been componentised so that they are based off a List of Rules and a Ranker whereby each Rule defines how a token(s) in a text can be matched to a semantic category. Given the matches from the Rules the for each token, a token can have zero or more matches, the Ranker ranks each match and finds the global best match for each token in the text. The taggers now support direct match and wildcard Multi Word Expressions. Due to this:
  • pymusas.taggers.rule_based.USASRuleBasedTagger has been changed and re-named to pymusas.taggers.rule_based.RuleBasedTagger and now only has a __call__ method.
  • pymusas.spacy_api.taggers.rule_based.USASRuleBasedTagger has been changed and re-named to pymusas.spacy_api.taggers.rule_based.RuleBasedTagger.
• A Rule system, of which all rules can be found in pymusas.taggers.rules:
  • pymusas.taggers.rules.rule.Rule an abstract class that describes how other sub-classes define the __call__ method and it's signature. This abstract class is sub-classed from pymusas.base.Serialise.
  • pymusas.taggers.rules.single_word.SingleWordRule a concrete sub-class of Rule for finding Single word lexicon entry matches.
  • pymusas.taggers.rules.mwe.MWERule a concrete sub-class of Rule for finding Multi Word Expression entry matches.
• A Ranking system, of which all of the components that are linked to ranking can be found in pymusas.rankers:
  • pymusas.rankers.ranking_meta_data.RankingMetaData describes a lexicon entry match, that are typically generated from pymusas.taggers.rules.rule.Rule classes being called. These matches indicate that some part of a text, one or more tokens, matches a lexicon entry whether that is a Multi Word Expression or single word lexicon.
  • pymusas.rankers.lexicon_entry.LexiconEntryRanker an abstract class that describes how other sub-classes should rank each token in the text and the expected output through the class's __call__ method. This abstract class is sub-classed from pymusas.base.Serialise.
  • pymusas.rankers.lexicon_entry.ContextualRuleBasedRanker a concrete sub-class of LexiconEntryRanker based off the ranking rules from Piao et al. 2003.
  • pymusas.rankers.lexical_match.LexicalMatch describes the lexical match within a pymusas.rankers.ranking_meta_data.RankingMetaData object.
• pymusas.utils.unique_pos_tags_in_lexicon_entry a function that given a lexicon entry, either Multi Word Expression or Single word, returns a Set[str] of unique POS tags in the lexicon entry.
• pymusas.utils.token_pos_tags_in_lexicon_entry a function that given a lexicon entry, either Multi Word Expression or Single word, yields a Tuple[str, str] of word and POS tag from the lexicon entry.
• A mapping from USAS core to Universal Part Of Speech (UPOS) tagset.
• A mapping from USAS core to basic CorCenCC POS tagset.
• A mapping from USAS core to Penn Chinese Treebank POS tagset tagset.
• pymusas.lexicon_collection.LexiconMetaData, object that contains all of the meta data about a single or Multi Word Expression lexicon entry.
• pymusas.lexicon_collection.LexiconType which describes the different types of single and Multi Word Expression (MWE) lexicon entires and templates that PyMUSAS uses or will use in the case of curly braces.
• The usage documentation, for the "How-to Tag Text", has been updated so that it includes an Indonesian example which does not use spaCy instead uses the Indonesian TreeTagger.
• spaCy registered functions for reading in a LexiconCollection or MWELexiconCollection from a TSV. These can be found in pymusas.spacy_api.lexicon_collection.
• spaCy registered functions for creating SingleWordRule and MWERule. These can be found in pymusas.spacy_api.taggers.rules.
• spaCy registered function for creating ContextualRuleBasedRanker. This can be found in pymusas.spacy_api.rankers.
• spaCy registered function for creating a List of Rules, this can be found here: pymusas.spacy_api.taggers.rules.rule_list.
• LexiconCollection and MWELexiconCollection open the TSV file downloaded through from_tsv method by default using utf-8 encoding.
• pymusas_rule_based_tagger is now a spacy registered factory by using an entry point.
• MWELexiconCollection warns users that it does not support curly braces MWE template expressions.
• All of the POS mappings can now be called through a spaCy registered function, all of these functions can be found in the pymusas.spacy_api.pos_mapper module.
• Updated the Introduction and How-to Tag Text usage documentation with the new updates that PyMUSAS now supports, e.g. MWE's. Also the How-to Tag Text is updated so that it uses the pre-configured spaCy components that have been created for each language, this spaCy components can be found and downloaded from the pymusas-models repository.

Removed

• pymusas.taggers.rule_based.USASRuleBasedTagger this is now replaced with pymusas.taggers.rule_based.RuleBasedTagger.
• pymusas.spacy_api.taggers.rule_based.USASRuleBasedTagger this is now replaced with pymusas.spacy_api.taggers.rule_based.RuleBasedTagger.
• Using PyMUSAS usage documentation page as it requires updating.

[codecov-commenter]

Codecov Report

Merging #32 (5feb6ef) into main (d381180) will increase coverage by 1.42%.
The diff coverage is 99.00%.

@@            Coverage Diff             @@
##             main      #32      +/-   ##
==========================================
+ Coverage   97.62%   99.05%   +1.42%     
==========================================
  Files           8       21      +13     
  Lines         337     1057     +720     
  Branches       66      214     +148     
==========================================
+ Hits          329     1047     +718     
+ Misses          7        0       -7     
- Partials        1       10       +9     

Impacted Files	Coverage Δ
pymusas/lexicon_collection.py	98.02% <97.49%> (-1.98%)	⬇️
pymusas/taggers/rules/mwe.py	98.73% <98.73%> (ø)
pymusas/rankers/lexicon_entry.py	99.18% <99.18%> (ø)
pymusas/__init__.py	100.00% <100.00%> (ø)
pymusas/base.py	100.00% <100.00%> (ø)
pymusas/pos_mapper.py	100.00% <100.00%> (ø)
pymusas/rankers/lexical_match.py	100.00% <100.00%> (ø)
pymusas/rankers/ranking_meta_data.py	100.00% <100.00%> (ø)
pymusas/spacy_api/lexicon_collection.py	100.00% <100.00%> (+100.00%)	⬆️
pymusas/spacy_api/pos_mapper.py	100.00% <100.00%> (ø)
... and 10 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dcd90e0...5feb6ef. Read the comment docs.