redact a pattern, but replace with a hash instead of a string

[tmccombs]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

I want to redact certain personally identifiable information, such as an email address, from logs, but in such a way that if I already know the information (email address), I can search the logs for matching log entries.

Attempted Solutions

I think it would be possible to do this with a lua transform. But the documentation for that says to create an issue if the remap transform doesn't meet my needs.

Lua also doesn't have a built in sha256 function so I would either need to use a lua native sha256 implementation, which I suspect would be slow, or pull in a shared library that adds such a function for lua (is that possible with vector?)

Proposal

I can think of a few ways this could be done:

• Add an option to the redact function that will replace the matched string with a hash of itself, Possibly with a configurable hash function, instead of with a fixed string.
• add a new function that replaces a matching regex with its hash
• make a function like replace, but instead of a string for the replacement, it uses a closure to compute a replacement value.

I think that the last bullet point is the most general and could be useful in other cases as well.

P.s. having a built in redaction pattern for email addresses would be awesome.

References

No response

Version

No response

[jszwedko]

Thanks for opening this @tmccombs ! This is something we considered when first implementing the redact function but sadly never circled back to add. It definitely makes sense (and we'd support a contribution here if motivated).

I also moved this issue to the VRL repo since the change would happen there.

[tmccombs]

Oh, haha, I already created a separate issue. I'll close this as a duplicate. Unless you think it would be worth having a function dedicated to hashing a pattern, in addition to or instead of a function to replace with a closure.

[jszwedko]

Apologies @tmccombs , The issue you linked is actually the same as this one 😄 I just moved it to this repo. I'll reopen it.

[tmccombs]

oops, I made the link wrong. I meant #628

[jszwedko]

Ah, I see. I think we can have both. This one can represent adding the ability to replace with a hash, easily. Even if we add support for replacing with the result of a closure I think having hashing be a first-class feature would be useful for discoverability and improved performance from a native implementation.

[tmccombs]

Sounds good to me. I'd be happy to make a PR for that as well, now that I am a little familiar with the codebase. Do you think it would be better to have a separate function (redact_hash maybe?) or add an optional option to the existing redact function?

[jszwedko]

That'd be great! Luckily the function implementations are relatively self-contained. Here is where redact is implemented: https://github.com/vectordotdev/vrl/blob/main/src/stdlib/redact.rs

I think adding additional configuration options to the redact function makes sense. For example, it could be called like:

redact("my id is 123456", filters: [r'\d+'], redactor: "sha256")

to replace 123456 with 8d969eef6ecad3c29a3a629280e686cf0c3f5d5a86aff3ca12020c923adc6c92. There was some discussion about this here and the original issue.