Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MapToList processor #3935

Closed
oeyh opened this issue Jan 9, 2024 · 2 comments · Fixed by #3945
Closed

MapToList processor #3935

oeyh opened this issue Jan 9, 2024 · 2 comments · Fixed by #3945
Assignees
Labels
plugin - processor A plugin to manipulate data in the data prepper pipeline.
Milestone

Comments

@oeyh
Copy link
Collaborator

oeyh commented Jan 9, 2024

Is your feature request related to a problem? Please describe.
A processor that converts a map of key-value pairs:

{
  "my-map": {
    "key1": "value1",
    "key2": "value2",
    "key3": "value3"
  }
}

to a list of object with key and value under separate fields:

{
  "my-list": [
    {
      "key": "key1",
      "value": "value1"
    },
    {
      "key": "key2",
      "value": "value2"
    },
    {
      "key": "key3",
      "value": "value3"
    }
  ]
}

This can be used when my-map contains a large number of fields to avoid mapping explosion in OpenSearch while keeping the key-value pairs searchable.

Describe the solution you'd like
This can be a reversed operation of what ListToMap processor does. The configuration options would be like this:

- map_to_list:
    source: "my-map"
    target: "my-list"
    key_name: "key"
    value_name: "value"

source(required): the source map to perform the operation
target(required): the target list
key_name: the key name of the field to hold the original key, default is "key"
value_name: the key name of the field to hold the original value, default is "value"

Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

@oeyh oeyh added the untriaged label Jan 9, 2024
@dlvenable dlvenable added plugin - processor A plugin to manipulate data in the data prepper pipeline. and removed untriaged labels Jan 9, 2024
@dlvenable dlvenable added this to the v2.7 milestone Jan 9, 2024
@eirsep
Copy link
Member

eirsep commented Jan 10, 2024

@kkondaka @oeyh @praveensameneni
The processor SHOULD NOT map all the fields into separate key and value object tuples

It should be able to accept a list of field name that should be skipped and retained in the "key":"value" pair format.

@oeyh
Copy link
Collaborator Author

oeyh commented Jan 10, 2024

Will add an exclude_keys option to skip the conversion for certain key-value pairs and also have an option to remove the converted fields from source map (retaining only the skipped fields in source map).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
plugin - processor A plugin to manipulate data in the data prepper pipeline.
Projects
Development

Successfully merging a pull request may close this issue.

3 participants