Ner model

[nsantacruz]

Overview

This PR in general adds functionality to the linker to (1) find named entities as well as citations (2) run the models in English.

The code needed to be refactored in order to make these changes in a clean way. Bugs were encountered with existing code and they were fixed as explained below.

Below is an overview of the major components that were changed.

Linker API

• Linker API now returns v3 results for English as well

Linker Python class

• Front door of linker is now Linker class instead of RefResolver.
• Linker can now detect both citations and named entities. Named entities are linked to Topics when possible
• Linker returns LinkedDoc which contains results for both named entities and citations

RefResolver

• RefResolver now has no access to ML models. It's sole responsibility is taking ML model results and resolving refs.
• Access to ML models is solely handled by NamedEntityRecognizer

NamedEntityRecognizer

• Takes input as a string and returns raw results from ML models wrapped in simple objects (RawNamedEntity or RawRef)

NamedEntityResolver

• Counterpart to RefResolver. Takes RawNamedEntity's and resolves them to ResolvedNamedEntity's

MapReferenceableBookNode

• ReferenceableBookNodes in general serve as a way of insulating all (or almost all) access to sefaria.model.schema to this layer of code. This makes the linker code much more decoupled from the rest of Sefaria-Project.
• The purpose of ReferenceableBookNodes are to map an Index schema to a tree structure that more accurately reflects a structure that is useful to the linker
• MapReferenceableBookNode corresponds to an ArrayMapNode in the schema.

AltStructNode

• Change the structural nodes of an alt struct (meaning, the nodes that are above the leaves) from TitledTreeNode to new class called AltStructNode
• This allows us to add some linker attributes to these nodes and find citations in alt structs

CORSDebugMiddleware

• It turns out this middleware is still useful when debugging the linker on localhost because nginx doesn't exist to add the CORS headers
  Linker JS
• Fix a bug that prevented citations from being wrapped when they appeared in their own block element (e.g.
  )

Linker Index Converter

• add more utilities to help convert indices to be usable by the Linker

Normalization

• normalization utilities are used by the linker to normalize input to the linker and map the results back onto the original indices of the text before normalization
• fixed some serious bugs that led to incorrect mappings from the normalized string to the original string
• add norm_to_unnorm_indices() which combines two commonly used functions into one.