NIFI-3055 StandardRecordWriter Can Throw UTFDataFormatException (1.x) #1469
Conversation
* Updated StandardRecordWriter, even though it is now deprecated to consider the encoding behavior of java.io.DataOutputStream.writeUTF() and truncate string values such that the UTF representation will not be longer than that DataOutputStream's 64K UTF format limit. * Updated the new SchemaRecordWriter class to similarly truncate long Strings that will be written as UTF. * Add tests to confirm handling of large UTF strings and various edge conditions of UTF string handling.
|
reviewing |
| final String truncated = utfString.substring(0, getCharsInUTFLength(utfString, MAX_ALLOWED_UTF_LENGTH)); | ||
| logger.warn("Truncating UTF value! Attempted to write string with char length {} and UTF length greater than " | ||
| + "supported maximum allowed ({}), truncating to char length {}.", | ||
| utfString.length(), MAX_ALLOWED_UTF_LENGTH, truncated.length()); |
There was a problem hiding this comment.
Can we mention provenance in this message, such as "Truncating provenance record value"? Does this message potentially mix char length and byte length, such as "Attempted to write string with char length 40000 and UTF length greater than supported maximum allowed (65535), truncating to char length 39000."? Perhaps a simpler message such as "Attempted to store string with length 40000, truncating to 39000."
* Clarify log message per code review comments.
|
@mosermw I have updated the log message in both PRs. Thanks! |
|
+1 looks good. I like the unit tests! I tested master branch before (exceptions galore and all partitions in flowfile repository was blacklisted) and after, we get appropriate WARN truncation messages in the logs. Will squash and merge. |
* Updated StandardRecordWriter, even though it is now deprecated to consider the encoding behavior of java.io.DataOutputStream.writeUTF() and truncate string values such that the UTF representation will not be longer than that DataOutputStream's 64K UTF format limit. * Updated the new SchemaRecordWriter class to similarly truncate long Strings that will be written as UTF. * Add tests to confirm handling of large UTF strings and various edge conditions of UTF string handling. Signed-off-by: Mike Moser <mosermw@apache.org> This closes apache#1469.
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:
Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically master)?
Is your initial contribution a single, squashed commit?
For code changes:
For documentation related changes:
Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.