Threshold selection tool

[samnlindsay]

Type of PR

• BUG
• FEAT
• MAINT
• DOC

Is your Pull Request linked to an existing Issue or Pull Request?

Relevant to #1564 in that it ties together several different charts into a single tool.

Creating draft PR while I gather feedback and then I'll add docs etc.

Do we want to replace the individual charts that this combines? Or add this as another function? Only it duplicates a lot of code and it's harder to maintain all of them, especially if they're not being used.

@ThomasHepworth FYI

Give a brief description for the solution you have provided

Combined interactive chart to:

• relate match weight threshold to a probability
• show the confusion matrix as a function of this threshold
• show the impact of threshold selection on performance metrics to aid threshold selection

Screenshot (using pairwise_labels.ipynb)

PR Checklist

• Added documentation for changes
• Added feature to example notebooks or tutorial (if appropriate)
• Added tests (if appropriate)
• Updated CHANGELOG.md (if appropriate)
• Made changes based off the latest version of Splink
• Run the linter

[ThomasHepworth]

Yah, really like this and it's essentially the direction I was going in.

I was leaning more towards simply combining the confusion matrix with the accuracy chart to reduce the amount of information on a user's screen:

(Ignore the lack of data in the accuracy chart, it was designed to simplify the json in the editor)

I can see the merit of showing the sigmoid'd match weight chart, so I am very much indifferent on this one.

I'll leave some additional thoughts tomorrow morning when I'm feeling a tad better.

[RobinL]

I don't feel at all strongly about it but I think I agree with Tom that the sigmoid doesn't add that much, especially if we used the waterfall trick of having two y axis (in this case it'd be the x axis) that map match weight and match probability on the main precision recall chart.

[samnlindsay]

Thanks both. I agree that from a purely information perspective the sigmoid doesn't add anything, but I think it helps make the information more accessible by facilitating different approaches to it. Whether you want to wiggle your cursor up and down the match probability axis or left and right across the match weight axis, it works equally well, while also literally drawing your attention to how your preferred measure maps onto the other.

For example:

• No intuition, naive approach - "match probability = 0.5 is the most sensible threshold right?".
• Partially informed approach - "The data linking team said cluster_medium was a threshold of 0.99. Is that really that different from 0.95?"
  • If what you know is probabilities, it's easier to navigate these charts using the 0-1 y-axis of the sigmoid.
• Maximum insight in minimal time approach - "What's the best this model can do? [goes to intersection of precision and recall curves - sees the sigmoid curve highlighted] Wow, that's pretty low!"
  • The visual element of seeing the probability close to 0 rather than 1 (for me anyway) makes the information much easier to digest than having to read off a weird probability scale like on the waterfall chart, or reading a value off a tooltip.

I've never liked that axis on the waterfall chart, but that's a rare example of something that only works in a match weight context, so the funky axis is the only way to visually cater for match probability.

[samnlindsay]

For comparison, here's how it might look without the sigmoid

[aalexandersson]

Is "score" a match probability? If yes, then it seems possible to optionally add back the sigmoid but still only use two charts (that is the confusion matrix plus one combined chart)!?

In the "Score" chart, the y-axis begins at 0.5. How does it look if it begins at 0? For titles, I suggest the y-axis title "Match probability (Score)" and the x-axis title "Match weight". Then, how does it look like to add (overlay) the sigmoid curve on this chart?

[RobinL]

I think the y axis 'score' refers to the 'precision or recall score'. (just 'score' because it's used for both metrics). But can see that's a potential source of confusion and perhaps we could find a better name ('performance metric score' perhaps?)

[aalexandersson]

The name 'performance metric score' is more general than 'precision or recall score', so better in that sense. My suggestion is to overlay (superimpose) the 'sigmoid' so that the result is one combined chart with two chart areas, not three chart areas. This way, we do require another chart area for the sigmoid.

[samnlindsay]

I experimented with being able to zoom in on the y ("Score") axis because the [0,1] range was too big to be useful, but I gave up because it was too difficult to constrain it the way I wanted (with 1 always the maximum).

The reason for the compromise shown here is that if precision/recall/F1 are below 0.5 your model isn't worth pursuing with. The area of interest is loosely described as "close to 1" in the score axis. In the example shown, the ideal range would be something like [0.85,1], and a [0,1] would contain a lot of whitespace and make it difficult to see the detail where needed.

Here's an example showing a less ambiguous y-axis title and with the full [0,1] range:

[aalexandersson]

Well, I think this latest combined chart is easier to interpret thanks to the full [0,1] range in the y-axis. For completeness, could you add back the sigmoid chart? That is, I would like to see the first chart but with modification of the full [0,1] range in the y-axis as in the latest chart.

[samnlindsay]

Following this discussion I have edited the chart as follows:

Given the "action" is all at the top end of the "Performance metric value" scale, I've left it at [0.5,1] to show more detail in the data.

[RobinL]

Should we remove linker.confusion_matrix_from_labels_table i.e. this
seems superfluous to have both ?

Given the functionality is so similar I wouldn't be worried about breaking backwards compatibility

[samnlindsay]

Yes, I was thinking about that in #2010

[RobinL]

@samnlindsay i edited the above because you referenced 2020, but i think you probably meant 2010

[samnlindsay]

Yes sorry I was on my phone

[RobinL]

No worries, just didn't want to edit without you knowing in case i did it wrong!

[RobinL]

The new code looks good to me.

Please can we remove linker.confusion_matrix_from_labels_table and associated code/docs as part of the PR

[samnlindsay]

confusion_matrix_from_labels_XXX functions and references to them have been removed.

The one exception is the blog post announcing the confusion matrix chart as a new feature. The image shown is unchanged but it now links to the new threshold selection tool notebook instead of the confusion matrix one.

[ThomasHepworth]

I think this may need to be changed to linker.threshold_selection_tool_from_labels_table(...)

[ThomasHepworth]

Sorry, a quick Q on this - should we set a default value for the match threshold selection?

Initially, the chart currently looks like so (at least for me):

where the confusion matrix doesn't appear until you hover over one of the graphs.

[aalexandersson]

I definitely would like to see the confusion matrix by default. It does not help to hover over one of the graphs when the output is PDF :-)