[HUDI-5829] Optimize conversion from json to row format when sanitizing field names#11941
Merged
jonvex merged 9 commits intoapache:masterfrom Oct 1, 2024
Merged
[HUDI-5829] Optimize conversion from json to row format when sanitizing field names#11941jonvex merged 9 commits intoapache:masterfrom
jonvex merged 9 commits intoapache:masterfrom
Conversation
vamsikarnika
force-pushed
the
json_to_row
branch
from
c86f236 to
1538426
Compare
vamsikarnika
changed the title
vamsikarnika
force-pushed
the
json_to_row
branch
from
e941544 to
a82fa20
Compare
added 2 commits
September 16, 2024 13:33
vamsikarnika
commented
Sep 16, 2024
Comment on lines
-313
to
-325
| // As we don't do rounding, the validation will enforce the scale part and the integer part are all within the | ||
| // limit. As a result, if scale is 2 precision is 5, we only allow 3 digits for the integer. | ||
| // Allowed: 123.45, 123, 0.12 | ||
| // Disallowed: 1234 (4 digit integer while the scale has already reserved 2 digit out of the 5 digit precision) | ||
| // 123456, 0.12345 | ||
| if (bigDecimal.scale() > decimalType.getScale() | ||
| || (bigDecimal.precision() - bigDecimal.scale()) > (decimalType.getPrecision() - decimalType.getScale())) { | ||
| // Correspond to case | ||
| // org.apache.avro.AvroTypeException: Cannot encode decimal with scale 5 as scale 2 without rounding. | ||
| // org.apache.avro.AvroTypeException: Cannot encode decimal with scale 3 as scale 2 without rounding | ||
| return Pair.of(false, null); | ||
| } | ||
|
|
Collaborator
Author
There was a problem hiding this comment.
Have moved this code to DecimalFieldProcessor class, so this can be reused by both JsonToAvro and JsonToRow processors.
vamsikarnika
commented
Sep 16, 2024
Comment on lines
+41
to
+46
| static Stream<Object> decimalBadCases() { | ||
| return Stream.of( | ||
| // Invalid schema definition. | ||
| Arguments.of(DECIMAL_AVRO_FILE_INVALID_PATH, "123.45", null, false), | ||
| // Schema set precision as 5, input overwhelmed the precision. | ||
| Arguments.of(DECIMAL_AVRO_FILE_PATH, "123333.45", null, false), |
Collaborator
Author
There was a problem hiding this comment.
Have moved the test data generators from TestMercifulJsonConvertor to this base class, so that both row and avro conversions can use same data for testing.
vamsikarnika
commented
Sep 16, 2024
Comment on lines
-76
to
+71
| @ValueSource(strings = { | ||
| "{\"first\":\"John\",\"last\":\"Smith\"}", | ||
| "[{\"first\":\"John\",\"last\":\"Smith\"}]", | ||
| "{\"first\":\"John\",\"last\":\"Smith\",\"suffix\":3}", | ||
| }) | ||
| @MethodSource("dataNestedJsonAsString") |
Collaborator
Author
There was a problem hiding this comment.
Have converted this MethodSource and moved it Test Base class to be reused by MercifulJsonToRowConverter
vamsikarnika
force-pushed
the
json_to_row
branch
from
2c511f1 to
857f45b
Compare
jonvex
requested changes
Sep 30, 2024
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/RowConverter.java
Outdated
Show resolved
Hide resolved
|
|
||
| import org.apache.hudi.exception.HoodieJsonConversionException; | ||
|
|
||
| public class HoodieJsonToRowConversionException extends HoodieJsonConversionException { |
Contributor
There was a problem hiding this comment.
Created https://issues.apache.org/jira/browse/HUDI-8279 as follow-up
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/RowConverter.java
Show resolved
Hide resolved
...ties/src/main/java/org/apache/hudi/utilities/sources/helpers/MercifulJsonToRowConverter.java
Outdated
Show resolved
Hide resolved
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/RowConverter.java
Show resolved
Hide resolved
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/RowConverter.java
Show resolved
Hide resolved
added 2 commits
October 1, 2024 17:36
vamsikarnika
force-pushed
the
json_to_row
branch
from
98f6dde to
ef80959
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Logs
Currently when source data has to read in row format and sanitization is enabled, we first read the data in avro format(which supports sanitization) and later convert from avro to row. This new approach simplifies this process by directly converting from json to row while applying sanitization.
Impact
When source data has to be read in row format, and sanitization is enabled. This change should make the conversion from json to row faster by directly converting from json to row.
This change directly affects the existing streams. This is currently added behind a flag which can be disabled if any issues are to be found.
Risk level (write none, low medium or high below)
None
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist