redesign when to generate ChunkGroupFooter #526
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## kill_thanos #526 +/- ##
==============================================
+ Coverage 41.6% 43.11% +1.5%
==============================================
Files 397 397
Lines 28193 28318 +125
Branches 4848 4889 +41
==============================================
+ Hits 11730 12208 +478
+ Misses 15403 14998 -405
- Partials 1060 1112 +52
Continue to review full report at Codecov.
|
liukun4515
requested changes
Dec 18, 2018
| @@ -78,10 +83,13 @@ public long getCurrentChunkGroupSize() { | |||
| return size; | |||
| } | |||
|
|
|||
| /** | |||
There was a problem hiding this comment.
I think the preFlush or the sealCurrentPage should not be public.
| * | ||
| */ | ||
| void preFlush(); | ||
| void sealCurrentPage(); |
There was a problem hiding this comment.
each chunkwriter should control package a page by itself when a page is full or the chunk will be flushed.
jixuan1989
added a commit
that referenced
this pull request
Jan 11, 2019
* add comment in PageHeader * add comment for TsFile, SeriesSchema, RecordUtils * add pagereader comment * add SeriesChunkReader comment * replace SeriesReader by TimeValuePairReader * add comment for class TsRowGroupBlockMetaData TsDeltaObject RowGroupMetaData * add comment for QueryConfig, QueryEngine, QueryType * translate a comment * format * change .ts to .tsfile * add comment for QueryDataSet, OnePassQueryDataSet, Field, OldRowRecord, RowRecord * recover ReadTest (not completed) and fix a neq filter bug * update ReadTest * fix a statistic bug and update ReadTest * update read demo * change RowGroupHeader to RowGroupFooter update ReadTest * add comment for IteratorOnePassQueryDataSet, TsPrimitiveType * add comment for IteratorOnePassQueryDataSet, TsPrimitiveType * add MetaMarker to distinguish RowGroupFooter and ChunkHeader * fix one SerializedSize bug add argument 'markerRead' in deserializeFrom() some refinements * fix one spelling miss * add comment for FilterUtils * to merge add_comment * add cov plugin and comment for encoding * update example * fix all reviews * fix reviews * move the project into subfolder * merge with master, and kill tsfile/interface * add second level index for tsfile * modify read process for usage of two level chunkgroup metadata * code format and rename RowGroup to ChunkGroup, SeriesChunk to Chunk, DeltaObject to Device * change delta object to device, row group to chunk group * change deltaFileWriter to fileWriter * fix ut test (for TsDeviceMetadataTest) * remove the .travis.yml * optimize ChunkLoaderImpl (#463) * add Metadata Seperator * remove hasNextChunkGroup in TsFileSequenceReader * add TsFileIOWriterTest * format * add reset() for decoder * delete * replace InputStream by ByteBuffer in PageDataReader (#467) * replace InputStream by ByteBuffer in PageDataReader * remove visitor * remove visitor in filter * change Long to long in filter * test * remove unnecessary T * fix issue 479：fix the bug caused by chunk cache loading (#481) * fix the bug of #479 The number of result is wrong when using filter, which is caused by chunk cache loading * replace readv2 to read, filterv2 to filter, and MeasurementDescriptor to MeasurementSchema in many classes' import clause. (#480) * recover example * change to long * remove satisfy in SeriesFilter * patch from write_thanos, for fix bug * rename reader * fix review * change tsfile io output as an interface (#493) * change tsfile io output as an interface * add new TsFileIoWriter for incomplete file * add method * add flush method to tsfileOutput * replace TimeValuePair by BatchData (#491) * read data in batch rather than one by one * discard TsPrimitive class * Timegenerator cache value optimization (#496) optimize time generator and hasNext & next * Move package (#497) * Move the read and write packages out of the timeseries package. * Rename class name *QueryFilter* to *Expression*. * Move the filter package into the read package. * Move the expression package out of the filter package. * Move all exception classes into cn.edu.tsinghua.tsfile.exception. * Update corresponding import in the IoTDB module. * Classify the common and utils packages. * fix the error in iotdb wal package; add processorexception * on the way * delete index package * remove package error * add utils * add TimeValuePair, format package * fix IExpression optimizer bug * remove fill and groupby package, format package * format package * comment old aggregate and query code * comment read analysis code * remove overflow update operation package * add transform filter operator to Expression * remove useless class in engine/control and fix pacakge in serialize * qp ok * QueryProcessExecutor ordinary query ok * format read * add series reader * modify read process * modify priority reader, merge reader and SealedReader, UnSealedReader * fix SealedTsFileReader, UnSealedTsFileReader * update priority merge reader * remove overflow chunk loader * delete QueryWithoutFilterExecutorImpl * format class code * format code * modify time generator * Engine data set without time generator * engine data set * replace FileChannel in tsfileSequenceReader as an interface * remove unused import * Write thanos (#514) * fix timegenerator and reader code * format code and doc * fix tsfile import * fix some import * remove query * add new class for reading unclosed TsFile * adapt UnSealedTsFileReader * update test in qp * refactor overflow and filenode * add fill * resolve the error of filter * delete generated codes by Thrfit * fix FileNodeProcessor * remove read package * fix test * delete interval tree test * resolve all error in filenode * fix error in tsfile metadata utils * add jdbc support * fix error in grafana * fix error in iotdb src/main/ * fix all error * modify the test for overflow and filenode * fix test error * delete some useless class * fix test * fix unit test * fix test error * modify the overflow test * fix test * delete fill feature * add IFill again * open limit and slimit in logical generator * fix plan test * format code * change some TsFileWriter functions as private; add comments in MemSeriesLazeMerger; fix the return value in write() in BufferWriteProcessor * fix bug when ReadWriteIOUtils.readAsPossible() read too much data. simplize read data by TsFileMetadataUtil in iotdb * change some parameter type from IMemSeries to TimeValuePairSorter if the parameter is only used for read * modifty some code for overflow test * better organize the IMemSeries and RawSeriesChunk (#522) * delete RawSeriesChunk, and change RawSeriesChunkLazyLoadImpl as ReadOnlyMemChunk * rename IMemSeries to IWritableMemSeries, and PrimitiveMemSeries to WriableMemChunk * remove the update and delete insert * add javadoc of BufferWriteProcessor * fix bug: add pre flush before write all chunk data into inputstream * make read program avaliable * add PriorityMergeReaderTest * fix the bug caused by querying empty series * modify the runing mode of test beautiful * redesign when to generate ChunkGroupFooter (#526) * fix read test * fix read bug * fix bug in read process * add override for some class * rename TestUitls to Constant * fix read test * fix problems due to alibaba java code * remove test in IoTDBCompleteTest * rename queryV2 package to query * change Action Map from <string, object> to <string, Action> * remove duplicated code in processor test * add javadoc for BufferWriteProcessor * add IoTDB query performance test * format code, fix bug * add TsFileDBConfig * fix too many open files bug of UnSeqFileReader * fix test * perfect test * fix query bug in UnSeqFile logic * fix test error * fix integration test error * submit performance test file, this shoule be deleted later * add ShowTimeseriesTest2 * perfect integration test * add MinMaxTime filter * remove performance package * add override method for some class * weaken the function of BufferWriteRestoreManager (#555) * add file stream manager to avoid too many oopen files error * add file stream manager for opened files * fix error in logback.xml configuration * Make TsFileSequenceReader concurrent safety. (#558) * add functions in TsFile reader for supporting concurrent reading (thread safe ) * remove commented codes, and prevent logback from outputting its own status at the start of every log * fix a bug * Fix unsafed serialized size #486 (#487) * fix #486 getSerializedSize() * fix null point * set reCalculateSerializedSize as private * Add timing opened file reader manager (#561) * merge master for clean ByteBuffer on high version JDK ( >1.8) * BufferWrite: Combine BufferWriteIO with BufferWriterRestoreManager into one class (#566) * delete SegmentInputStream and WithMMap * remove directByteBuffer check in TsFile, because we never use it * remove useless class (#570) * Replace delta object usage with device (#571) * remove tsfiledb to iotdb (#575) * merge with master (remove Chinese in the code)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fix issue #524 .
Then, there are two works left:
is the data size in
ChunkGroupFootermeaningful?if
MemtableFlushUtils.flushMemTable()needs to call low-level functions, I think theTsFileIOWriterneeds to be redesigned.I will try to fix the above two problem tomorrow.