Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New replace_keys Remap function #5377

Closed
bruceg opened this issue Dec 3, 2020 · 3 comments
Closed

New replace_keys Remap function #5377

bruceg opened this issue Dec 3, 2020 · 3 comments
Labels
domain: vrl Anything related to the Vector Remap Language needs: approval Needs review & approval before work can begin. sink: elasticsearch Anything `elasticsearch` sink related type: enhancement A value-adding code change that enhances its existing functionality.

Comments

@bruceg
Copy link
Member

bruceg commented Dec 3, 2020

Blocked in #5785. The remap language has support for manipulating strings in event values, for example with the replace function. To do the equivalent with keys is not possible.

The use case is described in #3588, where keys containing dots can lead to broken indexing. A trivial solution would be to replace all dots with underscores, as is done by the de_dot filter in both Elasticsearch and Logstash. Better would be to support fully parameterized replacement just like the replace function does for values.

Suggested function: replace_keys(map, regex | string, string, count = integer, depth = boolean) :: map
Parameters are the same as for replace except for the first and the return type, both of which are maps instead of strings, and the additional depth options, which will cause it to recurse into maps and replace all their keys as well.

Usage for the scenario above would be something like:
. = replace_keys(., ".", "_", depth = true)

@bruceg bruceg added type: enhancement A value-adding code change that enhances its existing functionality. domain: vrl Anything related to the Vector Remap Language labels Dec 3, 2020
@binarylogic binarylogic added the sink: elasticsearch Anything `elasticsearch` sink related label Dec 3, 2020
@jamtur01 jamtur01 removed this from the 2020-12-07: Nanite Repair System milestone Dec 5, 2020
@jszwedko
Copy link
Member

jszwedko commented Dec 8, 2020

Related: #4518

@binarylogic binarylogic added this to the 2021-01-04 Xenomass Well milestone Dec 26, 2020
@jamtur01 jamtur01 removed this from the 2021-01-04 Xenomass Well milestone Jan 1, 2021
@binarylogic binarylogic changed the title Add support for replacing keys to remap New replace_keys Remap function Jan 1, 2021
@binarylogic
Copy link
Contributor

I'm curious if we want to accept a value as the first argument here? Whenever we use:

. = ...

It is a flag to me that the function should operate on the entire event (like del). Otherwise, you could make the case that we should do:

. = del(.key)

As a first step, I think it's simpler to just make this an event-level function:

replace_keys('.', '_')

That example would replace all keys on the event.

@binarylogic binarylogic added the needs: approval Needs review & approval before work can begin. label Feb 21, 2021
@StephenWakely StephenWakely self-assigned this Mar 8, 2021
@StephenWakely StephenWakely removed their assignment Apr 1, 2021
@JeanMertz
Copy link
Contributor

JeanMertz commented Jun 7, 2022

Closing this, as we now have support for map_keys, which works similarly, but is more flexible:

 . = map_keys(object!(.), recursive: true) -> |key| { replace(key, ".", "_") }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: vrl Anything related to the Vector Remap Language needs: approval Needs review & approval before work can begin. sink: elasticsearch Anything `elasticsearch` sink related type: enhancement A value-adding code change that enhances its existing functionality.
Projects
None yet
Development

No branches or pull requests

6 participants