enhancement(vrl): add encode_key_value function

[prognant]

It reuses encode_logfmt and allows custom field & k/v delimiters.

encode_logfmt in now an alias with preset delimiters.

It can be use to construct datadog log tags list:

tags = parse_key_value!(.ddtags, field_delimiter: ",", key_value_delimiter: ":")
merge(tags,{"region": "ap-3", "via:vector" })
.ddtags = encode_key_value!(tags, key_value_delimiter: ":", field_delimiter: ",")

[spencergilbert]

Just one comment on the documentation side of things, otherwise looks good to me

[spencergilbert]

I wonder if that failed test is flakey 🤔

[prognant]

I wonder if that failed test is flakey 🤔

I think so as well, I believe that's not the first time it fails with no apparent reason or no obvious link to the code changed by the PR

[JeanMertz]

I left a few initial comments.

There are two "higher-level" thoughts that I have:

1. Do we want to specifically tie this to Datadog, or is this pattern (tags using foo:bar,baz:qux) generic enough that we can give it a more generic name?

2. If we do want to tie it to Datadog specifically, should we name it datadog_add_tags and datadog_remove_tags, similar to how we name all provider-specific parsing functions parse_<provider>_... (such as parse_aws_...)?

[prognant]

I left a few initial comments.

There are two "higher-level" thoughts that I have:

1. Do we want to specifically tie this to Datadog, or is this pattern (tags using foo:bar,baz:qux) generic enough that we can give it a more generic name?
2. If we do want to tie it to Datadog specifically, should we name it datadog_add_tags and datadog_remove_tags, similar to how we name all provider-specific parsing functions parse_<provider>_... (such as parse_aws_...)?

I think you made a point about being generic, we could assume that a tag list is generic enough (add key/value delimiter and and tags delimiter as an optional field) and instead have function named taglist_remove(...) and taglist_add(...) WDYT ?

[spencergilbert]

I think you made a point about being generic, we could assume that a tag list is generic enough (add key/value delimiter and and tags delimiter as an optional field) and instead have function named taglist_remove(...) and taglist_add(...) WDYT ?

Just noting that it seems similar to append with more structure

[JeanMertz]

Just noting that it seems similar to append with more structure

There's also parse_key_value which specifically allows specifying key_value_delimiter and field_delimiter, which seems even more closely related to these functions.

It appears this is identical to using add_datadog_tags and remove_datadog_tags:

$ tags = "foo:bar,baz:qux"
"foo:bar,baz:qux"

$ parsed_tags = parse_key_value!(tags, field_delimiter: ",", key_value_delimiter: ":")
{ "baz": "qux", "foo": "bar" }

$ del(parsed_tags.baz)
"qux"

$ parsed_tags.quux = "quuz"
"quuz"

$ parsed_tags
{ "foo": "bar", "quux": "quuz" }

Now, the thing that seems to be missing in VRL right now is "how to go from a k/v object to a k/v string?" We have convert functions for this. It seems to me what we really want in this case is to_key_value (or, to_key_value_string, to be more precise), which would behave something like this:

$ .ddtags = to_key_value(parsed_tags, field_delimiter: ",", key_value_delimiter: ":")
"foo:bar,quux:quuz"

---

Having said all this, I'm not against providing functions that make the above list of operations more convenient to perform, I'm just not entirely convinced the {add,remove}_datadog_tags functions meet that threshold (and if they do, I'm similarly not convinced they need to be Datadog-specific).

I'm happy to defer to others such as @jszwedko and @binarylogic to make that call.

[prognant]

I was unaware of the parse_key_value function, that's being say I'm now against my own PR...

Now, the thing that seems to be missing in VRL right now is "how to go from a k/v object to a k/v string?" We have convert functions for this. It seems to me what we really want in this case is to_key_value (or, to_key_value_string, to be more precise), which would behave something like this:

$ .ddtags = to_key_value(parsed_tags, field_delimiter: ",", key_value_delimiter: ":")
"foo:bar,quux:quuz"

Bouncing on that encode_logfmt would then be a specific case of to_key_value, with field_delimiter: " " and key_value_delimiter: "=", what I can think of right now would be to add a generic encode_key_value (to follow the existing function naming convention) that would do the same as your to_key_value suggestion, use the existing encode_logfmt code with slight changes and rewrite encode_logfmt as a simple wrapper. cc @JeanMertz

edit: I reworked the PR accordingly

[jszwedko]

One difference I'm noticing with parse_key_value is that it doesn't handle keys with no values. It seems like we support non-key:value tags in datadog? For example:

$ parse_key_value!("foo:bar,baz,host:localhost", field_delimiter: ",", key_value_delimiter: ":")
{ "baz,host": "localhost", "foo": "bar" }

We could modify parse_key_value to return something like:

{ "host": "localhost", "foo": "bar", "baz": null }

If I remember correctly, we also allow duplicate tags in Datadog which would throw another wrinkle into this. For example, what would foo:bar,foo:baz parse as?

I agree that these functions probably don't necessitate something Datadog specific but it also seems like parse_key_value/encode_key_value don't necessarily cover all of the cases.

That being said, I think this PR to add encode_key_value is fine to go in as it is a value-add regardless.

[prognant]

We could modify parse_key_value to return something like:

{ "host": "localhost", "foo": "bar", "baz": null }

It seems a suggestion to handle standalone key already exists out there https://pkg.go.dev/github.com/kr/logfmt?utm_source=godoc

Standalone key would end up being deserialised along with a true boolean for the matching value (instead of the null you suggested). I think it makes sense to use a boolean, the parse_key_value function would have no compatibility issue, the encode_key_value could have a extra settings to enable/disable standalone key serialisation for boolean.

As supporting standalone key (either as true or null for the value) seems doable without breaking any existing behaviour and also cover corner cases for datadog tags manipulation.

Edit: the latest commit support standalone key and should not break anything (maybe the verify parser also needs to ignore quoted string) cc @jszwedko

[prognant]

I split the PR in 2 different PR :

• This one about adding encode_key_value
• enhancement(vrl): standalone key support for parse_key_value #7907 about supporting standalone keys in parse_key_value

[prognant]

If I remember correctly, we also allow duplicate tags in Datadog which would throw another wrinkle into this. For example, what would foo:bar,foo:baz parse as?

I just checked, it works without any error, however the former is dropped silently and the latter is returned, a way to support same key tags would be to extend array processing function, a new function not really tied to datadog tag manipulation per se is available in #7908 could help doing some tag filtering just using join / split as simple parser/encoder.

[jszwedko]

Nice, thanks @prognant ! This looks good to me for adding the encode_key_value. I agree we could add an option to output keys with true values as just the key as a follow-up. This might be required for the datadog tags work?

Could we add this new function to the benches?

[JeanMertz]

Sweet! Almost there.

[JeanMertz]

Do we want to keep these (logfmt specific) tests, or were they moved to the new function?

[prognant]

I did the same as parse_logfmt/parse_key_value, i.e. no tests in the encode_logfmt (Existing ones were relocated in encode_key_value). I find it consistent, and as default separators match the logfmt format it is not illogical.

[JeanMertz]

It seems like we can make this function infallible if the fields_ordering argument is omitted?

[prognant]

I think so as well, I'll give it a try.

[prognant]

Nice, thanks @prognant ! This looks good to me for adding the encode_key_value. I agree we could add an option to output keys with true values as just the key as a follow-up. This might be required for the datadog tags work?

Could we add this new function to the benches?

Indeed, I was planning to work on that after getting that one merged, I think it makes sense to have encode_key_value and parse_key_value as symmetric as possible, and it would prevent datadog tags consisting in standalone key to be modified when edited using those function.

[jszwedko]

Nice!

[JeanMertz]

Two minor remarks, but otherwise this LGTM