Make blocking mechanism more efficient

[pudo]

The principal function of nomenklatura is to de-duplicate entity data, i.e. to find matches between different datasets (or within one dataset) and then merge multiple records in a useful way.

The first problem to solve on that path is to decide which entities might be the same, out of the full pool. If you think of two datasets sized 200,000 and 5,000,000 entities respectively, that's 1,000,000,000,000 comparisons. To avoid that, we have implemented a faster way to find potential matches.

This mechanism - blocking - works by basically constructing an in-memory search index of the whole dataset, where an entity {'id': 'Q7747', 'name': 'Vladimir Putin'}, gets turned into an inverted index ({'vladimir': ['Q7747'], 'putin': ['Q7747']} and then we generate pairwise scores of all entities by counting up how many index terms they have in common (while weighting the index terms using tf/idf). You can see this implemented here:

• https://github.com/opensanctions/nomenklatura/tree/master/nomenklatura/index
• In particular: https://github.com/opensanctions/nomenklatura/blob/master/nomenklatura/index/index.py#L143-L173

We need to improve the memory and footprint of doing this. I think it could be worthwhile exploring:

• Using numpy to store the inverted index and to generate the pairwise scoring as a numpy array
• Using more chaching in the index tokenizer to speed it up

For context: once the entities are blocked, a scoring mechanism is then used to decide which ones to either auto-match, or present to a user for arbitration.

[pudo]

@jbothma we could close this, too :)

[jbothma]

We've added an index class TantivyIndex using Tantivy for blocking #149