Preserve JSON types in flatten_json function #13947
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
patrickmann
force-pushed
the
preserveTypes
branch
from
9857c9c
to
6317c1f
Compare
mpfz0r
reviewed
Nov 14, 2022
| # Entry type according to https://keepachangelog.com/en/1.0.0/ | ||
| # One of: a(dded), c(hanged), d(eprecated), r(emoved), f(ixed), s(ecurity) | ||
| type = "fixed" | ||
| message = "Pipeline function flatten_json respects the original JSON types. An optional parameter is provided for back-compat." |
There was a problem hiding this comment.
Suggested change
| message = "Pipeline function flatten_json respects the original JSON types. An optional parameter is provided for back-compat." | |
| message = "The pipeline function flatten_json now respects the original JSON types. An optional parameter is provided for backwards compatibility" |
There was a problem hiding this comment.
I guess we should note that also in UPGRADING.md because it's a breaking change.
I do agree that we should change the default though, it's better to do it now, rather than to have the wrong default forever in our product.
There was a problem hiding this comment.
@patrickmann did you see my note about UPGRADING.md?
There was a problem hiding this comment.
Oops, missed that - fixed it
mpfz0r
reviewed
Nov 14, 2022
...og2-server/src/main/java/org/graylog/plugins/pipelineprocessor/functions/json/JsonUtils.java
Outdated
Show resolved
Hide resolved
mpfz0r
reviewed
Nov 14, 2022
...og2-server/src/main/java/org/graylog/plugins/pipelineprocessor/functions/json/JsonUtils.java
Outdated
Show resolved
Hide resolved
mpfz0r
reviewed
Nov 15, 2022
UPGRADING.md
Outdated
| @@ -43,6 +43,11 @@ It is now possible to change this behavior. When configuration property `stream_ | |||
| If all of your streams go to dedicated, separate index sets, it is advised to keep the default value of `stream_aware_field_types` property (`false`). It will decrease the load on ES/OS and stream separation across index sets already helps with showing proper fields for a query. | |||
| On the other hand, if multiple streams go to the same index sets, and you want precise field types and suggestions, you should set it to `true`. Consider monitoring your ES/OS load after that change, especially when using huge numbers of fields and streams. | |||
|
|
|||
| ## Breaking changes to pipeline functions | |||
There was a problem hiding this comment.
We already have a Breaking changes section in line 8
If I was to skim that document, I might have missed it down here.
mpfz0r
reviewed
Nov 15, 2022
mpfz0r
approved these changes
Nov 15, 2022
bernd
pushed a commit
that referenced
this pull request
Dec 9, 2022
* preserve JSON types when flattening * cl * add stringify optional param * make stringify parameter optional * update cl * incorporate review feedback * add to upgrading.md * put upgrade message in proper section * minor wording change
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #13888
Pipeline function
flatten_jsonreturns all of the extracted values as strings, regardless of original JSON type.We now honor the type.
See the associated issue for details and examples.