regex replace using a closure

[tmccombs]

I filed #633, but now realize that this might have been a better place to report it.

For convenience I will copy over my text from there:

Use Cases

I want to redact certain personally identifiable information, such as an email address, from logs, but in such a way that if I already know the information (email address), I can search the logs for matching log entries.

Attempted Solutions

I think it would be possible to do this with a lua transform. But the documentation for that says to create an issue if the remap transform doesn't meet my needs.

Lua also doesn't have a built in sha256 function so I would either need to use a lua native sha256 implementation, which I suspect would be slow, or pull in a shared library that adds such a function for lua (is that possible with vector?)

Proposal

I can think of a few ways this could be done:

• Add an option to the redact function that will replace the matched string with a hash of itself, Possibly with a configurable hash function, instead of with a fixed string.
• add a new function that replaces a matching regex with its hash
• make a function like replace, but instead of a string for the replacement, it uses a closure to compute a replacement value.

I think that the last bullet point is the most general and could be useful in other cases as well.

P.s. having a built in redaction pattern for email addresses would be awesome.

[tmccombs]

I don't currently have time to put together a full PR, but here is a rough draft of a partial implementation of a replace function that uses a closure:

fn replace_with<T>(value: Value, pattern: Value, ctx: &mut Context, runner: closure::Runner<T>) -> Resolved {
    let value = value.try_bytes_utf8_lossy()?;
    // TODO: support count?
    match pattern {
        Value::Regex(regex) => {
            let mut i: usize = 0;
            let mut failure: Option<ExpressionError> = None;

            let replaced = regex.replace_all(&value, |captures| {
                let captures_value = captures_to_value(captures);
                let result = runner.run_index_value(ctx, i, captures_value).and_then(|s| Ok(s.try_bytes_utf8_lossy()?));
                match result {
                    Ok(v) => v,
                    Err(e) => {
                        failure.get_or_insert(e);
                        Cow::from("")
                    }
                }
            }).as_bytes();
            if let Some(err) = failure {
                Err(e)
            } else {
                Ok(replaced.into())
            }
        }
        // TODO: should we also support Value::Bytes?
        value => Err(ValueError::Expected {
            got: value.kind(),
            expected: Kinde.regex()
        }.into())
    }
}

fn coaptures_to_value(captures: &Captures) -> Value {
    if captures.len() == 1 {
        // this is garanteed not to panic because the there is always 1 result.
        captures[0].into()
    } else {
        // return an array of the capture groups
        captures.iter().map(|m| m.as_str()).into()
    }
}

[pront]

This is a neat idea. If you are interested in driving this to completion, your draft is in the right direction, it just needs some boilerplate code, unit tests and a VRL test. Otherwise, we will add this to the backlog and prioritize accordingly.