kNN Classification: New meta scores/distances

[etiennedi]

This issue is meant to discuss new metrics to express how certain Weaviate is that something in a kNN classification was correctly classified.

@michaverhagen could you make a start by describing the ideas you already had? Thanks.

[michaverhagen]

• Winner's percentage: for example if k=7 and the 4 of the nearest neighbors have the same attribute (this making it the winning outcome), the winner's percentage would be 57%.
• Average winner's distance: take the average of the distance between the new concept and each of the (in the example 4) winning neighbors.

Ideally, what I would like to be able to do is the following:

• Assume we have a fully qualified training set
• Take a percentage of this training set, un-classify it, re-classify it and compare the new classification with the original one

Then using boundaries for the metrics, we can then have a discussion with customers: if we choose our metric at value X and assume everything with a score above X is classified correctly, then:

• 80% of the data is assumed to be classified correctly, which means that you would have to manually check 20% of your data. Is that acceptable?
• 98% of the 80% assumed correctly classified is in fact correct. Can you live with 2% wrongly classified data in your data.

[etiennedi]

Re winners percentage
Clear.

Re winning distances
Based on quickliy studying the code base:

It seems that the current winningDistance is already the mean of all winners, so what we don't currently have is the closestDistance. So, I could add that instead.

EDIT: Question was answered via Slack

Micha  3:31 PM
Do you mean closestDistance instead of numberOfNeighborsInWinningGroup (name tbd)?

etienne  3:32 PM
No, the numberOfNeighborsInWinningGroup you’ll get in any case

Micha  3:32 PM
Or do you mean all three:
winningPercentage
winningDistance = average of all winners
closestDistance

3:32
Ah ok perfect

etienne  3:33 PM
I investigated the current implemtation and realized that we already have winningDistance == average of all winners
3:33
exactly, so basically I’m proposing, since we already have winningDistance, we’ll add winningPercentage and closestDistance. Will that help or are we missing anything then?

Micha  3:34 PM
That would help greatly!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[michaverhagen]

This one is still relevant

[etiennedi]

prioritized for delivery before EOW (November 27).