enhancement(vrl): add filter_array

[prognant]

Allow to remove elements from an array based on a regexp.
Non-byte type are removed.

Note: as this PR was mostly anticipating a usecase (datadog tags manipulation) I suggest we don't merge it unless there is a real need for the usecase and enumeration/iteration is not available in VRL.

[JeanMertz]

I haven't reviewed the PR itself yet, but I did want to note that this gets close to a "wait until we have iteration" use-case (#6031).

That is to say, with iteration, presumably, filtering would be a matter of iterating and then filtering individual fields in an array or object, which would make a function like this obsolete.

I'm not against merging this, but I do think we need some form of iteration sooner or later, at which point we'll likely deprecate functions that like this one.

[jszwedko]

Very much agreed that this seems like a great case for iteration. I'm also ok with merging this if it is blocking other work, but it will (hopefully!) be deprecated shortly.

[pablosichert]

Looks good, @prognant!

Agree with @JeanMertz and @jszwedko regarding iteration.

Does this PR have a matching issue? If so, would be great to add it in the PR description as "Closes #XXX".

[pablosichert]

I'm wondering if we can restrict the input type further to only accept arrays of Value::Bytes?

[prognant]

I used the the same input kind as match_array, I don't think we can enforce that kind of restriction here, I guess we could do that in resolve but after having a quick look to existing function I did not find any previous similar implementation.

[pablosichert]

I'm not sure if I would expect this to match if the regex doesn't match exhaustively. Maybe we can add an option for that behavior?

[leebenson]

r'foo' matches foobar, so I'd expect the current behaviour.

[JeanMertz]

I think this is fine. We don't provide an exhaustive option elsewhere, and you can always use ^foo$ if need be.

[prognant]

Does this PR have a matching issue? If so, would be great to add it in the PR description as "Closes #XXX".

Not really, that work is related to easily enable datadog tags manipulation in VRL, e.g.:

tags = split(.ddtags, ",")
tags = filter_array(tags, r'(beta|staging)')
tags = push("via:vector", tags)
.ddtags = join(tags, ",")

That being said I agree that adding enumeration/looping/mapping would make this PR essentially useless.

[bits-bot]

All committers have signed the CLA.

[binarylogic]

@prognant what's the status here? Should we close since it seems to be stale?

[prognant]

@prognant what's the status here? Should we close since it seems to be stale?

This was a provision in case this was needed, as VRL iteration is planned, this is irrelevant.