feat(vrl): Support DataDog grok parser

[vladimir-dd]

This is the initial PR to support DD grok parsing.

Here is an example of grok rules:

access.common %{_client_ip} %{_ident} %{_auth} \[%{_date_access}\] "(?>%{_method} |)%{_url}(?> %{_version}|)" %{_status_code} (?>%{_bytes_written}|-)
access.combined %{access.common} (%{number:duration:scale(1000000000)} )?"%{_referer}" "%{_user_agent}"( "%{_x_forwarded_for}")?.*

You can write parsing rules with the %{MATCHER:EXTRACT:FILTER} syntax:

• Matcher: A rule (possibly a reference to another token rule) that describes what to expect (number, word, notSpace, etc.)
• Extract (optional): An identifier representing the capture destination for the piece of text matched by the Matcher.
• Filter (optional): A post-processor of the match to transform it.

Each rule can reference parsing rules defined above itself in the list. Only one can match any given log. The first one that matches, from top to bottom, is the one that does the parsing. For further documentation and the full list of available matcher and filters check out https://docs.datadoghq.com/logs/processing/parsing

We introduce a new parse_datadog_grok VRL function(hidden by FF). An example of usage:

parsed = parse_datadog_grok(value,  
 [
    'access.common %{_client_ip} %{_ident} %{_auth} \[%{_date_access}\] "(?>%{_method} |)%{_url}(?> %{_version}|)" %{_status_code} (?>%{_bytes_written}|-)',
    'access.combined %{access.common} (%{number:duration:scale(1000000000)} )?"%{_referer}" "%{_user_agent}"( "%{_x_forwarded_for}")?.*'
],
[
    '_auth %{notSpace:http.auth:nullIf("-")}',
    '_bytes_written %{integer:network.bytes_written}',
    '_client_ip %{ipOrHost:network.client.ip}',
    '_version HTTP\/(?<http.version>\d+\.\d+)',
    '_url %{notSpace:http.url}',
    '_ident %{notSpace:http.ident}',
    '_user_agent %{regex("[^\\\"]*"):http.useragent}',
    '_referer %{notSpace:http.referer}',
    '_status_code %{integer:http.status_code}',
    '_method %{word:http.method}',
    '_date_access %{date("dd/MMM/yyyy:HH:mm:ss Z"):date_access}',
    '_x_forwarded_for %{regex("[^\\\"]*"):http._x_forwarded_for:nullIf("-")}'
])

In the follow-up PRs we will add all missing matchers and filters.

Signed-off-by: Vladimir Zhuk vladimir.zhuk@datadoghq.com

[leebenson]

I'm still going through this, but wanted to drop a few early comments to be looking at in the meantime.

cargo check is also failing with 13 warnings, and cargo test with 7 errors.

Will continue going through 👍🏻

[leebenson]

Just a general curiosity, but I'm wondering if there was specific motivation for a home-brew version of query string parsing vs. reaching for something off-the-shelf, like serde_urlencoded?

[vladimir-dd]

Ah, good point, I believe something off-the-shelf could also work since DD support pretty standard behaviour. Will double-check and replace with the library one then.

[leebenson]

Has error 109 been documented anywhere? I noticed https://errors.vrl.dev jumps from 108 to 110.

[vladimir-dd]

Hm, nice catch, I copied it from 'parse_grok'. @StephenWakely(as the author of the "standard" function) - do you think we need to document this error message?

[leebenson]

Looks great @vladimir-dd 🎉

Left a few comments, but nothing I'd consider blocking.

[netlify]

❌ Deploy Preview for vector-project failed.

🔨 Explore the source changes: 5cbe134

🔍 Inspect the deploy log: https://app.netlify.com/sites/vector-project/deploys/6116694aee0656000898ed7f

[StephenWakely]

It is possible for a static expression to be types other than a string (numbers, timestamps) so try_bytes could well fail here. The error needs to be propagated up so that VRL can report it properly. expect will cause a panic.

It is fiddly with chained functions, but something along the lines of this should do it:

                    .map(|e| {
                        Ok(e.try_bytes_utf8_lossy()?
                            .into_owned())
                    })
                    .transpose()?

[StephenWakely]

As above.

[StephenWakely]

It would be useful to document what the parameters prev_rule_definitions and prev_rule_filters are used for. It's not instantly apparent to me.

[vladimir-dd]

Yeah, definitely. Refactored and added some comments in 5f3fc10

[StephenWakely]

Suggested change

	for r in raw_grok_patterns {
	for (r, pure) in raw_grok_patterns.iter().zip(pure_grok_patterns.iter()) {

[vladimir-dd]

Fixed in d924c53

[StephenWakely]

Pattern matching may be nicer here.

Suggested change

	rule.destination.is_some() && rule.destination.as_ref().unwrap().filter_fn.is_some()
	matches!(
	rule,
	GrokPattern {
	destination: Some(Destination {
	filter_fn: Some(_),
	..
	}),
	..
	}
	)

[vladimir-dd]

Yes, definitely, the final version is in dce9511

[StephenWakely]

But given all that, I think this whole chain could be reduced to a single loop which means it could avoid having to create this intermediate Vec.

[vladimir-dd]

Indeed! Simplified in 9cd2497.

[binarylogic]

Just to bring visibility to discussions we've had across the team, people are having difficulty reviewing this PR, as evident by the number of comments (75+). I hate to say this, but I think we should close this PR and identify incremental changes that can be reviewed appropriately and quickly. I'm open to pushing this over the finish line, but historically that has not worked out well.

[jszwedko]

@vladimir-dd feel free to flag whenever this is ready for review again.

[vladimir-dd]

Closing this PR to break it down in smaller chunks. Will address all left comments in the follow-up PRs.