New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a length limit of 512 to the properties stored in the segment metadata #6008
Add a length limit of 512 to the properties stored in the segment metadata #6008
Conversation
362fa39
to
5593656
Compare
Codecov Report
@@ Coverage Diff @@
## master #6008 +/- ##
===========================================
- Coverage 66.44% 46.34% -20.11%
===========================================
Files 1075 1180 +105
Lines 54773 55889 +1116
Branches 8168 8134 -34
===========================================
- Hits 36396 25899 -10497
- Misses 15700 27869 +12169
+ Partials 2677 2121 -556
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
@@ -78,6 +78,9 @@ | |||
public class SegmentColumnarIndexCreator implements SegmentCreator { | |||
// TODO Refactor class name to match interface name | |||
private static final Logger LOGGER = LoggerFactory.getLogger(SegmentColumnarIndexCreator.class); | |||
// Allow at most 512 characters for the metadata property | |||
private static final int METADATA_PROPERTY_LENGTH_LIMIT = 512; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shall this align with FieldSpec.DEFAULT_MAX_LENGTH
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They are independent, where the maxLength
is configurable in the FieldSpec
to allow ingesting long strings, but the property length limit is fixed. Also, the property length limit also applies to the BYTES column.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. Thanks
Description
Prevent storing very long values into the segment metadata. This could happen for text column, bytes column for serialized objects, or blob store use cases (not recommended but supported).