feat: new split_to_map udf #5563
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
agavra
reviewed
Jun 8, 2020
ksqldb-engine/src/main/java/io/confluent/ksql/function/udf/string/SplitToMap.java
Show resolved
Hide resolved
| + "'kvDelimiter'. If the same key is present multiple times in the input, the latest " | ||
| + "value for that key is returned. Returns NULL f the input text or either of the " | ||
| + "delimiters is NULL.") | ||
| public class SplitToMap { |
There was a problem hiding this comment.
We may also want to have EncodeMap(map, entry_delim, kv_delim) which encodes a map into a string
There was a problem hiding this comment.
i'll add it to my udf backlog ;) - although i'm less concerned about that from a use-case perspective as you can always construct almost-arbitrary json output by using structs/maps/arrays if you need that for a downstream system. The primary motivator for this one is when you get, for example, some encoded message from a mainframe MQ system that needs to be parsed out this way
JimGalasyn
reviewed
Jun 8, 2020
| ``` | ||
|
|
||
| Splits a string into key-value pairs and creates a map from them. The | ||
| 'entryDelimiter' splits the string into key-value pairs which are then split by 'kvDelimiter'. If the same key is present multiple times in the input, the latest value for that key is returned. |
There was a problem hiding this comment.
Suggested change
| 'entryDelimiter' splits the string into key-value pairs which are then split by 'kvDelimiter'. If the same key is present multiple times in the input, the latest value for that key is returned. | |
| `entryDelimiter` splits the string into key-value pairs which are then split by `kvDelimiter`. If the same key is present multiple times in the input, the latest value for that key is returned. |
JimGalasyn
reviewed
Jun 8, 2020
| Splits a string into key-value pairs and creates a map from them. The | ||
| 'entryDelimiter' splits the string into key-value pairs which are then split by 'kvDelimiter'. If the same key is present multiple times in the input, the latest value for that key is returned. | ||
|
|
||
| Returns NULL f the input text is NULL. |
There was a problem hiding this comment.
Suggested change
| Returns NULL f the input text is NULL. | |
| Returns NULL if the input text is NULL. |
There was a problem hiding this comment.
arghhh! thanks Jim :)
JimGalasyn
approved these changes
Jun 8, 2020
agavra
approved these changes
Jun 10, 2020
| import java.util.Map; | ||
| import org.junit.Test; | ||
|
|
||
| public class SplitToMapTest { |
There was a problem hiding this comment.
can you add a test with whitespace? what do we want the behavior to be when there is whitespace (e.g. foo := bar)
There was a problem hiding this comment.
good idea, test added!
agavra
approved these changes
Jun 11, 2020
agavra
pushed a commit
that referenced
this pull request
Jun 11, 2020
New UDF split_to_map(input, entryDelimiter, kvDelimiter) to build a map from a string. Useful for taking messages from upstream systems and converting them into a more structured and usable format.
JimGalasyn
added a commit
that referenced
this pull request
Jun 25, 2020
* feat: implements ARRAY_JOIN as requested in (#5028) (#5474) (#5638) Co-authored-by: Hans-Peter Grahsl <hpgrahsl@users.noreply.github.com> * feat: new split_to_map udf (#5563) New UDF split_to_map(input, entryDelimiter, kvDelimiter) to build a map from a string. Useful for taking messages from upstream systems and converting them into a more structured and usable format. * feat: add CHR UDF (#5559) A new UDF, CHR, to turn a number representing a unicode codepoint into a single-character string. Very useful for dealing with non-printable characters (tab, CR, LF, ...) in strings or those characters not easily represented in your local codepage. Co-authored-by: Steven Zhang <35498506+stevenpyzhang@users.noreply.github.com> Co-authored-by: Hans-Peter Grahsl <hpgrahsl@users.noreply.github.com> Co-authored-by: Nick Dearden <blueedgenick@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
New UDF
split_to_map(input, entryDelimiter, kvDelimiter)to build a map from a string.Useful for taking messages from upstream systems and converting them into a more structured and usable format.
Testing done
New Unit & QTT tests.
Reviewer checklist