Dealing with "PII bleed" -- errors that correlate to another record

[William Navarre (wnavarre)]

In the dataset I'm encountering situations where there is "PII bleed."

That is, the data isn't typo'd in some random way, but rather is correlated with another record.

As an example, two people with radically different names have the same email, phone number, address, and even the same date of birth!

In certain cases it's obvious the email belongs to one of the names and not the other. You can imagine a data entry person probably was sloppy and didn't reset the form between inputting data or something.

If it weren't effecting date-of-birth, then a multi-stage approach would make sense. But since it is, this is a more severe error imo.

How do folks generally deal with this issue? So far when I've identified it I've tried to exclude that data from training and that's about it -- but maybe someone has thought of a more clever solution?

[Mike Edwards (medwar99)]

Definitely a tricky scenario.

To play devil's advocate on your example:

• I could legally change my first and last name and still retain all my other information, both those records should belong to me as a real-world entity
• I could be sharing an email/phone/address with my differently-name twin and the records should not be linked

How would you differentiate that genuine true match against a false match?

Similar scenarios exist on the other fields:

• "Same fields but different DOB" - Is this John Smith Sr and John Smith Jr living at the same address, or one John Smith with an error in DOB?
• "Same fields but different address" - Has John Smith moved, or is this two John Smiths in different places?

The FS model assumes independence between information, and Splink is scoring based on comparison buckets (exact match, distance etc.) rather than the specific values themselves (ignoring term frequency adjustments). If you want to factor in the correlation between values then you could do some feature engineering to combine field information for scoring, e.g. name || email might allow the model to tease apart johnsmith_johnsmith@email.com and janedoe_johnsmith@email.com. This would incorporate some of that information into the model, rather than discarding it.

Ultimately it comes down to reasoning why it may be an error vs bleed. This can include how the data was collected, the purpose of the collection, mechanisms of validation and verification etc. It also depends on your tolerance to that kind of "erroneous" link/non-link in your use case, whether you are pushing for precision or recall, availability of contextual information, linkage quality reporting practices, and whether you have ability to do some post-processing on the linked data before its use (e.g. dropping clusters that fall short of some business rules).

[William Navarre (wnavarre)]

I think my goal is:

• At training time, try to isolate these issues with as narrowly-tailored a solution as possible for the m training.

• At inference time/post-processing, try to detect these situations.

The training symptom I'm trying to get rid of is where "total mismatch on name" gets an insufficiently negative weight because EM sees too many apparent matches where name fits in that "worst" comparison level.

Currently I do that by just excluding the data source that has this problem the worst from the EM training pairs. This helps a lot, but is a very NOT narrowly tailored solution.

[William Navarre (wnavarre)]

Also, without revealing too much about my data comes from...

We have certain data that comes in in batches. Often times, that batch is of size 1 (probably 90% of the time or so), but can just as well be of size 2 or 3.

One source of the PII bleed tends to be within these batches.

Thinking about that:

• Same date of birth, e.g., is STRONGER if both of the persons are the only guy in their batch.

• Same date of birth, e.g., is significantly WEAKER if both of the persons have that same DOB.

Like if the "batch" is Joe Smith and John Allen, and they both are supposedly born October 15, 1970 -- there's a much higher probability that for one of them it's wrong than if they were in their own "batch."

I'll consider adding extra match levels for this kind of thing. But I want to be careful because this itself might create spurious correlation / unintended consequences I think. 👎