A multi-lingual suite for named-entity recognition in Python.
NERwhal's mission is to make defining custom recognizers for different NER approaches as easy as possible. To achieve this, different NER backends are implemented behind a unified API. Each recognizer is based on one of the backends. Users can detect named entities by implementing custom recognizers for one or more of the backends.
Check out our blog post about NERwhal on Medium.
NERwhal makes use of some of the most powerful NER platforms out there:
- Regular expressions: Using regular expressions you can define a named entity as a set of strings.
- Entity Ruler: spaCy’s Entity Ruler lets you define patterns for sequences of tokens. (spaCy is also used for tokenization)
- FlashText: The FlashText Algorithm can search texts very efficiently for long lists of keywords.
- Deep Learning: The Stanza library and models (which provide state-of-the-art results for NER in many languages) power NERwhal's statistical recognition. Currently, Stanza supports NER for 8 languages.
The suite can combine the results of these methods in a smart way to get best results. E.g. a match with a higher score can overwrite a lower scored one, or, if one entity was identified several times, its confidence score can be increased.
Each recognizer can define a list of context words that may occur in the context of named entities. If a context word is found in the same sentence as the entity, the confidence score is increased.
NERwhal follows the philosophy that recognizers are specific to the language, use case, and requirements. The recommended way to use is to define your own custom recognizers. Yet to exemplify its usage and to help you bootstrap your own recognition suite, some example recognizers are implemented in nerwhal/integrated_recognizers. Please refer to each recognizers' PyDoc for more information, and keep in mind that none of these recognizers will catch all occurrences of their category, and that they may produce false positives results.
NERwhal can be installed from PyPi and has to be installed in a virtual environment (venv or conda for instance).
pip install nerwhalTo recognize named entities, pass a text and config object to the recognize method.
Select the recognizers to be used in the config object.
>>> from nerwhal import recognize, Config
>>>
>>> config = Config(language="de", use_statistical_ner=True, recognizer_paths=["nerwhal/integrated_recognizers/email_recognizer.py"])
>>>
>>> recognize("Ich heiße Luke und meine E-Mail ist luke@skywalker.com.", config=config, return_tokens=True)
{
'tokens': [
Token(text='Ich', has_ws=True, br_count=0, start_char=0, end_char=3),
Token(text='heiße', has_ws=True, br_count=0, start_char=4, end_char=9),
...
Token(text='.', has_ws=False, br_count=0, start_char=54, end_char=55)
],
'ents': [
NamedEntity(start_char=10, end_char=14, tag='PER', text='Luke', score=0.8, recognizer='StanzaNerBackend', start_tok=2, end_tok=3),
NamedEntity(start_char=36, end_char=54, tag='EMAIL', text='luke@skywalker.com', score=0.95, recognizer='EmailRecognizer', start_tok=7, end_tok=8)
]
}To implement a custom recognizer, you have to implement one of the interfaces in recognizer_bases. For examples see one of the integrated_recognizers.
You can install all (production and development) requirements using:
pip install -r requirements.txt
This repository uses git hooks to validate code quality and formatting.
pre-commit install
git config --bool flake8.strict true # Makes the commit fail if flake8 reports an error
To run the hooks:
pre-commit run --all-files
Run all tests with:
pytest --cov-report term --cov=nerwhal
To skip tests that require the download of Stanza models run:
pytest -m "not stanza"
For usage questions, bugs, or suggestions please file a Github issue. If you would like to contribute or have other questions please email hello@openredact.org.