Redact Ingest Processor

[davidkyle]

The redact processor uses the Grok rules engine to redact matching Grok patterns. Existing patterns from the Grok pattern bank can be referenced directly and new patterns added inline in the pattern_definitions option.

One application of the redact processor is to obscure Personal Identifying Information by configuring the processor to detect known patterns such as email or IP addresses.

In an ingest pipeline the redact processor could be augmented by a Named Entity Recognition model to detect and remove names, places.

PUT _ingest/pipeline/redact
{
  "processors": [
    {
      "redact": {
        "field": "to_redact",
        "patterns": ["%{EMAILADDRESS:EMAIL}", "%{IP:IP_ADDRESS}", "%{CREDIT_CARD:CREDIT_CARD}"],
        "pattern_definitions": {
          "CREDIT_CARD": "\\d{4}[ -]\\d{4}[ -]\\d{4}[ -]\\d{4}"
        }
      }
    }
  ]
}

Given an input document with the field to_redact

{
  "to_redact": test@elastic.co sent an email from the IP 192.168.0.1."
}

The redact processor, as configured above, will emit

{
  "to_redact": <EMAIL> sent an email from the IP <IP_ADDRESS>."
}

The matched text is replaced by the Grok pattern name. The < and > tokens surrounding the replaced text are configurable via the prefix and suffix options.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[elasticsearchmachine]

Hi @davidkyle, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-docs (Team:Docs)

[joegallo]

I'm not sure whether we should be invoking createGrokThreadWatchdog twice, or merely creating a single value once and passing it to both the grok and redact processor factories. I'll figure out the answer and let you know.

[davidkyle]

Good catch! The MatchWatchdog is used by all the Grok processors on this node and all the Groks that those processors create, it makes sense to use the same watchdog for the Groks created by the redact processor.

I pushed 940c3c1

[joegallo]

New patterns can be defined with a regular expression or combine Grok patterns from the base definitions to build complex patterns.

This sentence seems a bit stilted to me.

[davidkyle]

👍 Updated with an example of defining a custom pattern

[joegallo]

re: setCurrentClass(className) -- is className the best name for what this is (the variable, the setter, the field of the extractor that the setter is controlling)? Perhaps this is better as patternName throughout? (I noticed a comment where you referenced it as "Grok pattern name".) I'm open to other ideas here, but I don't love className (there's a bit of a garden path problem with 'classes' and 'class names' on the JVM having a particular meaning).

Anyway, open to ideas, or counter arguments.

[davidkyle]

patternName is better, I've used that consistently throughout now

[joegallo]

Generally I think this looks great, please take my comments as relatively minor notes along the way towards me getting to a ✅.

[abdonpijpelink]

LGTM from a docs perspective. I've left some minor suggestions.

[joegallo]

Great feature! Thanks for writing it.