Entity service only requesting the top 5 matches for each chunks comparison

[mpnd]

I'm concerned with the following line on compute_filter_similarity() in async_worker.py:

chunk_results = anonlink.entitymatch.calculate_filter_similarity(chunk_dp1, chunk_dp2,
                                                                 threshold=threshold,
                                                                 k=5,
                                                                 use_python=False)

Why is k arbitrarily set to 5? Is there a better value for k and why?

[hardbyte]

We can probably get rid of k entirely - or handle None so we return all matches over threshold.

[unzvfu]

The function anonlink.entitymatch.calculate_filter_similarity has a Python implementation and a C implementation; the choice being determined by the parameter use_python. For some reason the Python version doesn't use k or threshold at all.

[unzvfu]

The primary purpose of both k and threshold seems to be to prune the number of relevant entries in the similarity matrix; threshold being a kind of soft limit and k being a hard limit. Is that right? If so, the correct value of k could just be some system parameter that's determined by how much memory we have available.

[unzvfu]

As mentioned in data61/anonlink#56 the default values of k and threshold vary from function to function which suggests that maybe there are no suitable default values. If not, then the defaults should be removed everywhere and the caller should be obliged to specify them each time.

[unzvfu]

After discussion with @hardbyte we are leaning towards removing k altogether---essentially setting it equal to n---, as (i) it is often set to n in practice and (ii) we don't have a very clear example of a case where setting it is an obviously good idea or net win. The main argument for keeping k (indeed the only one that I can see) seems to be that we can use it to reduce the memory footprint when we are confident that the data is clean enough that the correct match will be within the top k scores.

[unzvfu]

I think this is closed by #84. Some related discussion is at anonlink/issues/56.