Skip to content

add more testing for MV var-length columns, detect overflow scenarios#8286

Merged
richardstartin merged 1 commit intoapache:masterfrom
richardstartin:mv-strings-overflow
Mar 3, 2022
Merged

add more testing for MV var-length columns, detect overflow scenarios#8286
richardstartin merged 1 commit intoapache:masterfrom
richardstartin:mv-strings-overflow

Conversation

@richardstartin
Copy link
Copy Markdown
Member

This addresses a bug report of a buffer overflow creating MV string raw forward index, with the following stack trace:

Start building IndexCreator!
Could not build segment
java.lang.IllegalArgumentException: newPosition > limit: (1048580 > 1048575)
	at java.nio.Buffer.createPositionException(Buffer.java:318) ~[?:?]
	at java.nio.Buffer.position(Buffer.java:293) ~[?:?]
	at java.nio.ByteBuffer.position(ByteBuffer.java:1094) ~[?:?]
	at java.nio.MappedByteBuffer.position(MappedByteBuffer.java:226) ~[?:?]
	at java.nio.MappedByteBuffer.position(MappedByteBuffer.java:67) ~[?:?]
	at org.apache.pinot.segment.local.io.writer.impl.VarByteChunkSVForwardIndexWriter.putStrings(VarByteChunkSVForwardIndexWriter.java:119) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
	at org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueVarByteRawIndexCreator.putStringMV(MultiValueVarByteRawIndexCreator.java:106) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.indexRow(SegmentColumnarIndexCreator.java:543) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:244) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
	at org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:131) ~[pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:815) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:746) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:644) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-bb98ed89e7e2f06138139f58ec0ec6d4f9944a16]
	at java.lang.Thread.run(Thread.java:829) [?:?]

I have added extensive testing for the impacted components and could not find any buffer overflows, and have narrowed this down to an integer overflow such that the product of Integer.BYTES + maxRowLengthInBytes + maxLengthPrefixes and a fixed numDocsPerChunk overflows and is non-negative. Very large MV values is a limitation of the current implementation and I have added precondition checks to detect overflow unambiguously.

@codecov-commenter
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.77%. Comparing base (e498a71) to head (e32266e).
⚠️ Report is 6312 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #8286      +/-   ##
============================================
- Coverage     70.82%   70.77%   -0.05%     
- Complexity     4239     4243       +4     
============================================
  Files          1631     1631              
  Lines         85461    85463       +2     
  Branches      12877    12879       +2     
============================================
- Hits          60527    60486      -41     
- Misses        20749    20804      +55     
+ Partials       4185     4173      -12     
Flag Coverage Δ
integration1 28.88% <0.00%> (-0.11%) ⬇️
integration2 27.58% <0.00%> (-0.02%) ⬇️
unittests1 66.92% <100.00%> (+<0.01%) ⬆️
unittests2 14.16% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing Related to tests or test infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants