feat(new transform): New explode transform

[eunchuldev]

Ref: #6459

Signed-off-by: Eunchul Song eunchulsong9@gmail.com

[jszwedko]

This is great work, thanks @sech9446 ! Thank you for adding all of the tests.

I left a few small comments. I'm also wondering if we might be able to expand the functionality of this transform to handle arrays of objects differently.

I'm imagining a case where a user might have input like:

{
  "events": [
    {
      "message": "event 1"
    },
    {
      "message": "event 2"
    }
  ]
}

And they want to output:

{
  "message": "event 1"
}

and

{
  "message": "event 2"
}

I think this could be handled by checking the type of the array value as we are iterating and, for objects, instead of inserting as a child field, merge with the original event.

I could see this being a configurable option so that users could still have it as a child field when desired, which is how you currently have it.

This will also need docs, but I can help with that once it is ready to be merged.

[jszwedko]

I think you can just pull the serde(default) up to the struct level.

[jszwedko]

For other parsing transforms, like json_parser, we've typically included a drop_invalid config option that users could use to either drop events that don't match expectations (if true) or pass them along as-is (if false). Could we add that option here as well? We've typically defaulted to false which would pass along non-confirming events as-is.

[jszwedko]

Suggested change

	message = "Cannot explode non array kind field.",
	message = "Cannot explode non-array field.",

[jszwedko]

I think you could just use .get() here to avoid the unnecessary insert operation.

[eunchuldev]

Thank you for the detailed review!

Your suggestion makes sense. I am going to add two new configurable options drop_invalid and flatten, with default value false of both.

[jszwedko]

@sech9446 thanks for your work on this! We used it as inspiration for the unnest function in #7404 which, combined with #7267 will let users do this transformation just in remap. I'll close this out given the other approach, but let us know if that approach doesn't work for you.