[SPARK-40625] Add MASK_CCN and TRY_MASK_CCN functions to redact credit card string values

[dtenedor]

What changes were proposed in this pull request?

Add MASK_CCN and TRY_MASK_CCN functions to redact credit card string values.

Each of these functions converts a string 'input' representing a credit card number to an updated version applying a transformation to the characters. This can be useful for creating copies of tables with sensitive information removed, but retaining the same schema.

Both functions return an error if the format string is invalid.

MASK_CCN returns an error if the input string does not match the format string.

TRY_MASK_CCN instead returns NULL in that case.

The format can consist of the following characters, case insensitive:

• Each 'X' represents a digit which will be converted to 'X' in the result.
• Each digit '0'-'9' represents a digit which will be left unchanged in the result.
• Each '-' character should match exactly in the input string.
• The default format string is: XXXX-XXXX-XXXX-XXXX.

No other format characters are allowed.
Any leading or trailing whitespace in the input string is stripped out.

Examples:

> SELECT MASK_CCN(ccn) FROM VALUES ("1234-5678-9876-5432") AS tab(ccn);
  XXXX-XXXX-XXXX-XXXX
> SELECT MASK_CCN("  1234-5678-9876-5432  ", "XXXX-XXXX-XXXX-1234");
  XXXX-XXXX-XXXX-5432
> SELECT MASK_CCN("[1234-5678-9876-5432]", "[XXXX-XXXX-XXXX-1234]");
  Error: the format string is invalid
> SELECT MASK_CCN("1234567898765432");
  Error: the input string does not match the format
> SELECT MASK_CCN("1234567898765432", "XXXX-XXXX-XXXX-1234");
  Error: the input string does not match the format
> SELECT TRY_MASK_CCN("1234567898765432");
  NULL
> SELECT TRY_MASK_CCN("1234567898765432", "XXXX-XXXX-XXXX-1234");
  NULL

Why are the changes needed?

These functions are useful for processing string values.

Does this PR introduce any user-facing change?

Yes, it adds new SQL functions.

How was this patch tested?

This PR adds a new unit test suite.

[AmplabJenkins]

Can one of the admins verify this patch?

[dtenedor]

Hi @vitaliili-db can you please help review this when you have time? 🙏

[dtenedor]

@vitaliili-db @amaliujia thanks so much for your review! I responded to your comments, please take another look.

[amaliujia]

LGTM with one comment

[dtenedor]

The only test failure is unrelated and flaky

[gengliangwang]

Shouldn't we check if the corresponding format string char is space as well?

[dtenedor]

Good question, I think the intention is to skip and ignore whitespace in both the input string and the format string. For example, for use cases like credit card numbers, phone numbers, and social security numbers, these are not material to the data in the field. I put a comment to this effect here.

[dtenedor]

Update on this: per our offline conversation, we decided to require whitespace in the format string to match the input string exactly. I updated the PR accordingly.

[gengliangwang]

Seems we are going to skip the spaces on both input/format string...

[dtenedor]

Same, this is per intention (left a comment).

[dtenedor]

Update on this: per our offline conversation, we decided to require whitespace in the format string to match the input string exactly. I updated the PR accordingly.

[dtenedor]

@gengliangwang thanks for review! 👍 Responded to your comments.

[dtenedor]

@gengliangwang @vinodkc FYI, it looks like there is a duplication of planned effort between this PR and https://issues.apache.org/jira/browse/SPARK-40686. We should probably dedup these into one effort. @vinodkc do you want to take this on, and I can close my Jira and help with the review?

[dtenedor]

going to close this in favor of the other work by @vinodkc instead