[To rel/1.1]Fix WrappedSegment extension with big name on edge case#9484
Merged
MarcosZyk merged 2 commits intoapache:rel/1.1from Mar 31, 2023
Merged
[To rel/1.1]Fix WrappedSegment extension with big name on edge case#9484MarcosZyk merged 2 commits intoapache:rel/1.1from
MarcosZyk merged 2 commits intoapache:rel/1.1from
Conversation
MarcosZyk
approved these changes
Mar 30, 2023
wangchao316
requested changes
Mar 30, 2023
server/src/test/java/org/apache/iotdb/db/metadata/mtree/schemafile/SchemaFileTest.java
Outdated
Show resolved
Hide resolved
server/src/test/java/org/apache/iotdb/db/metadata/mtree/schemafile/SchemaFileTest.java
Outdated
Show resolved
Hide resolved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
In determining the size of the pre-allocated segment for an EntityNode, an accurate calculation is performed by examining the children of the EntityNode if it has fewer than 20 children. Alternatively, the size is estimated using a hard-coded step list based on the cardinality of the EntityNode's children.
During the flushing process of the EntityNode, the segment size is assessed prior to the insertion of each child, and the segment is re-allocated or extended according to the anticipated size. If the segment is unable to accommodate the inserted child even after an extension, an exception will be raised, indicating that the record is too large or the extension has failed.
This issue arises in an edge case where the EntityNode has fewer than 20 children, each with a large size (approximately 1000 bytes in total) when calculated for pre-allocation. However, before the EntityNode is flushed, a new child with a large name and a small buffer, which includes aliases and other schema data but measures under 20 bytes, for instance, is inserted. The segment size assessment may be misled, resulting in a minor increment for the segment extension, e.g., from 1000 bytes to 1022 bytes, leading to an overflow since the segment cannot accommodate the large name.
This hotfix takes into account the length of the name, bytes of the name, and the record buffer offset when assessing the expected size of the WrappedSegment. This issue will not occur with AliasIndexPage or InternalPage, as their lengths will never be assessed multiple times due to their fixed length. The root cause of the original issue, as well as potential performance improvements, can be attributed to the design of the WrappedSegment increment, which is determined by a hard-coded step list as aforementioned.