Ethics in Data Linking

[zslade]

Hi folks! 👋

The Splink team have recently been exploring the subject of ethics in data linking 🔗 .

Data linking generally happens at the start of analytical projects so any decisions on model design, or biases introduced, will have consequences for all downstream use cases of that data. With this in mind, it is important to try and address any potential ethical issues at the linking stage.

Sharing both our current knowledge and future discoveries on the ethics of data linking with Splink is important to help support our users and the data linking community more broadly. We've written a blog post as a first step on that journey and to kick-start a conversation with our users on ethics, so we'd love to hear your thoughts!

• Did you find this blog post useful?
• Would you like to see more posts like this in the future?
• What ethical issues have you come across in your own linkage projects? How have you approached them?
• How could Splink better support you in tackling ethical issues? What do you think of our future plans, as laid out in the blog post?

[aalexandersson]

Thanks for the great blog post! Here are two suggestions for improvement:

1. Patient and data protection in reporting: Several of the Splink charts, for example the waterfall chart, can display highly sensitive patient data. In the example of the waterfall chart, the tutorial mentions the DOB and email of "Grace Kelly". If it would be real data, such charts generally should not be used in reporting, I would love to have a simple way to report the waterfall chart and other charts but without the patient-level data for better protection.
2. Communication about restricted documentation: For example, PR #1478 New link accuracy chart links to the document "Performance Metrics for Linking" which I very much appreciated at the time but I just noticed that the document is no longer public. In the example, why is the document now restricted? Can it be released again in some way? Performance metrics are critical (💯). Therefore, in general, please do not restrict previously released documentation even if it is just work in progress.

[aalexandersson]

@RossKen Thank you!

1. No that's it 👍. Anonymisation of the waterfall chart would be especially beneficial. In general, "reporting" charts such as the waterfall chart would benefit more from anonymisation than "pre-processing" charts such as the profile columns chart.

2. The main thing to improve in the documentation, in my view, is easier reproducible code for the tutorials. In addition, the tutorial introduction ends with a link to example notebooks which currently gives a "404 - Not found" error.

In terms of the evaluation guidance documentation that you are working on, I think it would be helpful to consider common problems. I recommend providing an example of how to construct the truth table when there is missing data. when either the actual match status data are missing for the linked data or the linked data are missing for the actual match status data. Most performance metrics tutorials instead assume complete data, which is not realistic. See my related unanswered discussion issue #1884.

[RossKen]

1. Super - we will get on that over the next few weeks.
2. Thanks for the broken link feedback - I have raised a PR to fix it. Could you maybe give some more detail on what you mean by easier, reproducible code? Is this separate to your comments further down on the missing data and the impact on performance metrics?

On the common problems, we know that missing data is something we need to improve on so this is helpful for us to prioritise. How we create and use labelled data is a broader issue that we are also looking to make progress on over the coming months.

[aalexandersson]

Thanks again. This is great support.

By "easier reproducible code for the tutorials" I simply mean to have one single file with the code. The tutorial has 7 steps with codes in steps 2-7 which are six separate web pages: from 02_Exploratory_analysis.html to 07_Quality_assurance.html. If I want to actually reproduce the results, it seems that I will have to copy-paste the code from those six web pages. It would be a bit easier to instead only copy or use one single file with the code. Something similar exists for the Examples in the folder demos/examples. It would be a simple fix to add a perhaps one sentence with a link to such self-contained code.

Yes, having better user documentation on labelled data and on the related manual review tool would help me to better understand how Splink constructs the truth table and the confusion matrix when there is missing data.

Update: The missing data can be either in the larger dataset or in the smaller dataset, and either in the predicted data or in the actual data with the true match status. Here is a similar discussion for scikit-learn but it only discusses missingness in the predicted data.

[RossKen]

Oh, I hadn't thought about that before - great suggestion! It is on my list to go and do a full update of the tutorial over the next few months to integrate the latest feature so I can do this at the same time.

[RossKen]

Hey @aalexandersson FYI I have just raised a PR for the anonymised Waterfall chart - #1938