[To dev/1.3] Fix compression by type 1.3 by jt2594838 · Pull Request #16218 · apache/iotdb

jt2594838 · 2025-08-21T01:45:05Z

…15118) (#15123) * finish * fix IT * fix IT --------- Signed-off-by: OneSizeFitQuorum <tanxinyu@apache.org>

…memchunk (#15115) * perf: batch iterator for multiple TVLists/alignedTVList * fix bug * current time * single TVList/alignedTVList iterator * fix: queryAlignChuckWithDeletionTest * fix: pr review * fix: test cases * remove clone in sortTvListForFlush

… longer acquire lock

…15168) Co-authored-by: luoluoyuyu <zhenyu@apache.org> Co-authored-by: Steve Yurong Su <rong@apache.org>

…#15165) (#15166)

…5158) (#15162) * stash for debug * fix comparator (#15157) LGTM!!! * adjust logs --------- Co-authored-by: Li Yu Heng <liyuheng55555@126.com>

…15131) (cherry picked from commit 50a48ce)

…#15179) (#15184) Co-authored-by: Steve Yurong Su <rong@apache.org> (cherry picked from commit bc078fc)

…pling of prefetched logs (#15175) (#15187)

(cherry picked from commit 299c3df)

(cherry picked from commit a81c5c6)

…15170) * fix: MemoryNotEnough exception when flushing try to release tvlist * retry release tvlist if memory not enough during flush * synchronized clone method for TVList * query context list -> query context set * fix: remove query context when abort finish * refactor: reorganize AbstractWritableMemChunk * fix: float/double value * fix: some testcase bugs * fix: delete column bug for aligned timeseries * fix: clone data types list when create new AlignedTVList * fix: bitmap is null when delete column * init bitmaps when it is null * format * fix: error from cherry-pick * set memchunk datatypes when handover

(cherry picked from commit ebb1bcf)

…15226) Co-authored-by: Steve Yurong Su <rong@apache.org>

… tsfile batch to avoid cleaning before ack (#15229) (#15233)

…t Mod file after the Mod file was cleaned up after the Receiver thread exited. (#15235)

…emory reservation logic of tsFile parser provider & Subscription IT: assertGte for received tsfile count (#15068) (#15070) Co-authored-by: VGalaxies <vgalaxies@apache.org>

…15047) Co-authored-by: Steve Yurong Su <rong@apache.org>

#15202) & Pipe: Prevent Duplicate Time Usage Reporting on Close in PipeTsFileToTabletsMetrics (#15220) (#15224) Co-authored-by: Steve Yurong Su <rong@apache.org>

…ransferring tsfiles & Fix validateTsFile and shouldMarkAsPipeRequest may not be effective (#15245)

…etection (#15210) (#15246) Co-authored-by: Steve Yurong Su <rong@apache.org> (cherry picked from commit 8889255)

…Pipe: Fixed the client release problem in trackable handler (#16130) * Temporarily banned some PBTree IT to avoid CI failure (#16127) * PBTree-fix * revert * npe

#16008)

…llow up fix for #16008) (#16136)

…on (#16110) (#16129) * Skip TTL check when there is no ttl in the data region and no mods file * check back quote

* Load: Reduce unnecessary MD5 calculations * spotless (cherry picked from commit 5a9f628)

…) (#16150) * test * test * fix * fix

#16152)

…mproved the handling logic for rowCount and null value bitmaps in insertTabletNode (#16133)

…isabled (#16162)

…16145) (#16157) * Pipe: Optimized the commit queue to avoid OOM problem (#16145) * first * trial * mayfix * may_final * may_last * may_fix * partial * completion * enhance-test * fix * bug-fix-n * test * test

…t enough

… by parallel dispatching

…16M (#16174) (#16185)

#16197) (#16200) * Pipe: Optimized the memory usage of progress report event (#16197) * may_complete * simplify * Update PipeHistoricalDataRegionTsFileSource.java

…erAutoCreate with batch mode (#16202) (#16204) * Pipe CI: always flush for IoTDBPipeDataSinkIT.testReceiverAutoCreate with batch mode * fixup

(cherry picked from commit cdb3599)

* Pipe: Optimize TsFile parsing logic * update (cherry picked from commit 0dc583e)

…mpactionPerformer (#16183) * Using PatternTreeMap to cache mod entries in ReadChunkCompactionPerformer * modify SettleSelector * fix * fix

…e judgment (#16212) * optimize * shit * test * revert * opti * fix

VGalaxies and others added 30 commits March 18, 2025 16:50

Subscription IT: fix IoTDBConsumer2With1TopicShareProcessTsfileIT (#1…

0361309

…5110) (#15121)

Make removed confignode can rejoin cluster without deleting data dir (#…

c2ffa8e

…15118) (#15123) * finish * fix IT * fix IT --------- Signed-off-by: OneSizeFitQuorum <tanxinyu@apache.org>

[To dev/1.3] Optimize query execution in easy query case (#15128)

658c267

[To dev/1.3] Enhance remove unexisted datanode error message #15140

85d435b

[To dev/1.3] Print debug log only if debug is enbale

0355587

[to dev/1.3] RegionMigrateProcedure and RegionReconstructProcedure no…

7bb1049

… longer acquire lock

Updated moquette version to fix MQTT write direct memory leak issue (#…

d49f43a

…15168) Co-authored-by: luoluoyuyu <zhenyu@apache.org> Co-authored-by: Steve Yurong Su <rong@apache.org>

Pipe: Fixed the bug that OPC DA sink may free a string multiple times (…

ecb500e

…#15165) (#15166)

Fix bug that the partition table auto cleaner cannot be activated (#1…

af472c0

…5158) (#15162) * stash for debug * fix comparator (#15157) LGTM!!! * adjust logs --------- Co-authored-by: Li Yu Heng <liyuheng55555@126.com>

[To dev/1.3] Restore rollback for UnsetTemplateProcedure (#15088)

960f00b

[To dev/1.3] Update tsFile snapshot version

d8acf63

Pipe: Fixed the ttl privilege logic at receiver side (#15135) (#15145)

c1ab85f

Load & Region Migrate: Notify all DNs before and after RM (#15032) (#…

04b5ba4

…15131) (cherry picked from commit 50a48ce)

Pipe: Fixed Async Connector Repeatedly Putting Events into RetryQueue (…

8080d1f

…#15179) (#15184) Co-authored-by: Steve Yurong Su <rong@apache.org> (cherry picked from commit bc078fc)

Fix division by zero error when flush memtable (#15183)

78dfd29

Subscription: intro SubscriptionLogManager for global uniform downsam…

50cc5da

…pling of prefetched logs (#15175) (#15187)

Improve ConfigNode removing #15189 (#15195)

0042ab9

(cherry picked from commit 299c3df)

Pipe: iotdb-thrift-connector async retry mechanism (#14916) (#15201)

9f277c0

Load: Support multi-disks awareness for IoTV2 and Pipe (#15181) (#15213)

2594187

(cherry picked from commit a81c5c6)

Load: Add default value for multi disks awareness (#15216) (#15218)

a9c82fb

(cherry picked from commit ebb1bcf)

Pipe: Enable showing WRITE_BACK_SINK (Fix IoTDBDatabaseIT) (#15209) (#…

a629754

…15226) Co-authored-by: Steve Yurong Su <rong@apache.org>

Subscription: distinguish between reference count of ack and clean in…

f2029f8

… tsfile batch to avoid cleaning before ack (#15229) (#15233)

Pipe: Fixed the issue where the storage engine referenced an incorrec…

860dd1c

…t Mod file after the Mod file was cleaned up after the Receiver thread exited. (#15235)

Pipe: Reduced the log of epoch switching & Refactor & Optimized the m…

2b7f32d

…emory reservation logic of tsFile parser provider & Subscription IT: assertGte for received tsfile count (#15068) (#15070) Co-authored-by: VGalaxies <vgalaxies@apache.org>

Pipe: Added memory control for receiver insert statements (#15046) (#…

dce37ef

…15047) Co-authored-by: Steve Yurong Su <rong@apache.org>

Pipe: Add metrics for tsfile to tablets invocation call count and time (

6c92f63

#15202) & Pipe: Prevent Duplicate Time Usage Reporting on Close in PipeTsFileToTabletsMetrics (#15220) (#15224) Co-authored-by: Steve Yurong Su <rong@apache.org>

Pipe: Fix stuck caused by async connector client not returned after t…

399ba48

…ransferring tsfiles & Fix validateTsFile and shouldMarkAsPipeRequest may not be effective (#15245)

Load: Implement region operations cache for load × region migration d…

f8b368b

…etection (#15210) (#15246) Co-authored-by: Steve Yurong Su <rong@apache.org> (cherry picked from commit 8889255)

Caideyipi and others added 28 commits August 8, 2025 17:30

[To dev/1.3] Temporarily banned some PBTree IT to avoid CI failure & …

d37e9aa

…Pipe: Fixed the client release problem in trackable handler (#16130) * Temporarily banned some PBTree IT to avoid CI failure (#16127) * PBTree-fix * revert * npe

Pipe: Avoid unnecessary close-client in async client (Follow up fix for

92368e6

#16008)

[To dev/1.3] Pipe: Avoid unnecessary close-client in async client (Fo…

46b69a7

…llow up fix for #16008) (#16136)

Skip TTL check when there is no TTL and no mods file in the data regi…

31a407a

…on (#16110) (#16129) * Skip TTL check when there is no ttl in the data region and no mods file * check back quote

[To dev/1.3] Load: Reduce unnecessary MD5 calculations (#16141) (#16151)

f3fa09f

* Load: Reduce unnecessary MD5 calculations * spotless (cherry picked from commit 5a9f628)

[To dev/1.3] Pipe: Use fixed memory settings to configure batch (#16134…

b9b9119

…) (#16150) * test * test * fix * fix

[To dev/1.3] Load: Fix LoadPieceNode reading aligned Chunk causing OOM (

436dd37

#16152)

[To dev/1.3] Pipe: Removed the useless ser-de in receiver raw req & I…

f5f52a4

…mproved the handling logic for rowCount and null value bitmaps in insertTabletNode (#16133)

[To dev/1.3] Pipe: Changed the default periodical login behavior to d…

6f8efa9

…isabled (#16162)

[To dev/1.3] Pipe: Optimized the commit queue to avoid OOM problem (#…

e102e33

…16145) (#16157) * Pipe: Optimized the commit queue to avoid OOM problem (#16145) * first * trial * mayfix * may_final * may_last * may_fix * partial * completion * enhance-test * fix * bug-fix-n * test * test

Replace the key in FileReaderManager to TsFileID

1a3f960

[To dev/1.3] Pipe: Delete unnecessary metrics (#16165) (#16168)

e9039a7

[To dev/1.3] Fix nodeUrls shuffle error (#16167)

08b3150

[to dev/1.3] Fix the status code of fetching schema when memory is no…

3b9513e

…t enough

Fix concurrent modification of non-thread-safe data structures caused…

ef872f3

… by parallel dispatching

[To dev/1.3] Use oracle jdk in 1c3d IT (#16182)

489bd73

[to dev/1.3] Add memory control for mod entries in query

5d9d7ae

[To dev/1.3] Pipe: Changed back the default value of tsFile batch to …

7082d00

…16M (#16174) (#16185)

[To dev/1.3] Pipe: Optimized the memory usage of progress report event (

35039c5

#16197) (#16200) * Pipe: Optimized the memory usage of progress report event (#16197) * may_complete * simplify * Update PipeHistoricalDataRegionTsFileSource.java

[To dev/1.3] Pipe CI: always flush for IoTDBPipeDataSinkIT.testReceiv…

1ac6984

…erAutoCreate with batch mode (#16202) (#16204) * Pipe CI: always flush for IoTDBPipeDataSinkIT.testReceiverAutoCreate with batch mode * fixup

[To dev/1.3] Pipe: Comment Config Check (#16194) (#16208)

4592368

(cherry picked from commit cdb3599)

[To dev/1.3] Pipe: Optimize TsFile parsing logic (#16173) (#16207)

bee05d1

* Pipe: Optimize TsFile parsing logic * update (cherry picked from commit 0dc583e)

[to dev/1.3] Using PatternTreeMap to cache mod entries in ReadChunkCo…

04b7e23

…mpactionPerformer (#16183) * Using PatternTreeMap to cache mod entries in ReadChunkCompactionPerformer * modify SettleSelector * fix * fix

Load: Optimized the partial path split logic in modifications coverag…

f51a15f

…e judgment (#16212) * optimize * shit * test * revert * opti * fix

Print measurement for putting buffer exceptions (#15873) (#16213)

e0a90fb

Add configs to set default compression for each data type (#16117)

ad064d8

Fix that compression by type is not properly applied (#16211)

f52f5f0

Fix some capitalizations in config template (#16216)

17456c0

jt2594838 closed this Aug 21, 2025

jt2594838 deleted the fix_compression_by_type_1.3 branch August 21, 2025 03:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[To dev/1.3] Fix compression by type 1.3#16218

[To dev/1.3] Fix compression by type 1.3#16218
jt2594838 wants to merge 811 commits intomasterfrom
fix_compression_by_type_1.3

jt2594838 commented Aug 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Comments

Conversation

jt2594838 commented Aug 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Comments