Tvlist feat new by shizy818 · Pull Request #14616 · apache/iotdb

shizy818 · 2025-01-02T12:40:04Z

Description

Content1 ...

Content2 ...

Content3 ...

This PR has:

Key changed/added classes (or packages if there are too many classes) in this PR

* out of mempage bounds check * overlapped data error during query

* change some list to array * remember row count in tvlist iterator

* memory control of column access

...rc/main/java/org/apache/iotdb/db/queryengine/execution/fragment/FragmentInstanceContext.java

...e/src/main/java/org/apache/iotdb/db/schemaengine/schemaregion/utils/ResourceByPathUtils.java

...main/java/org/apache/iotdb/db/storageengine/dataregion/memtable/AlignedWritableMemChunk.java

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/datastructure/TVList.java

iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties.template

...de/src/main/java/org/apache/iotdb/db/utils/datastructure/MergeSortAlignedTVListIterator.java

* bitmap mark * sequence row count

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java

shizy818 · 2025-01-17T08:21:26Z

Write Performance Test

# iotdb settings
series_slot_num=100
data_region_per_data_node=5

# benchmark settings
DEVICE_NUMBER=100
SENSOR_NUMBER=20
LOOP=10000
DATA_CLIENT_NUMBER=10
DATA_CLIENT_NUMBER=10
OPERATION_PROPORTION=1:0:0:0:0:0:0:0:0:0:0:0

Unaligned Series:

TVList Sort Threshold	0	100	200	500	1000	2000	5000	10000	master
points/s	4569824.325	4374989.505	4342791.53	4481212.23	4335337.15	4217159.025	4333778.515	4191477.875	4355698.645
Comparison	1.049159893	1.004428878	0.997036729	1.028815948	0.995325321	0.968193479	0.994967482	0.96229749

Aligned Series:

TVList Sort Threshold	0	100	200	500	1000	2000	5000	10000	master
points/s	6845973	6616038.8	7266610.87	7332207.695	6747288.985	6925542.685	7476763.485	7219287.525	7161518.61
Comparison	0.955938729	0.923831824	1.014674578	1.023834203	0.942158968	0.967049457	1.044019277	1.008066573

shizy818 · 2025-01-17T08:40:19Z

Write/Query mixed mode

# iotdb settings
series_slot_num=100
data_region_per_data_node=5

# benchmark settings
DEVICE_NUMBER=100
SENSOR_NUMBER=20
LOOP=10000
DATA_CLIENT_NUMBER=10
DATA_CLIENT_NUMBER=8
IS_RECENT_QUERY=true
OPERATION_PROPORTION=1:0:1:1:1:1:1:1:0:1:1:1

Unaligned Series:

TVList Sort Threshold	0	100	200	500	1000	2000	5000	10000	master
INGESTION	1960317.5	1943008.65	2019711.86	2072910.49	2051436.53	2059098.48	1922076.65	1835176.4	1394176.65
PRECISE_POINT	0		0	0	0	0	0	0	0
TIME_RANGE	1480.28	1373.54	1399.18	1417.37	1461.29	1449.54	1394.38	1374.51	919.82
VALUE_RANGE	1483.19	1412.85	1414.09	1438.75	1470.86	1437.06	1421.54	1390.75	894.92
AGG_RANGE	78.6	77.91	80.98	83.11	82.25	82.56	77.07	73.58	55.9
AGG_VALUE	77.45	76.77	79.8	81.9	81.05	81.35	75.94	72.51	55.08
AGG_RANGE_VALUE	76.88	76.21	79.21	81.3	80.46	80.76	75.38	71.98	54.68
GROUP_BY	1032.57	1023.45	1063.85	1091.87	1080.56	1084.6	1012.42	966.65	734.36
LATEST_POINT	0	0	0	0	0	0	0	0	0
RANGE_QUERY_DESC	1471.21	1379.31	1387.84	1441.23	1454.83	1446.81	1396.66	1370.4	906.93
VALUE_RANGE_QUERY_DESC	1476.39	1392.44	1387.69	1425.76	1467.74	1438.82	1392.74	1378.52	906.6
GROUP_BY_DESC	1010.02	1001.1	1040.62	1068.03	1056.96	1060.91	990.31	945.54	718.32
Comparison	1.406086424	1.39366856	1.448685822	1.486844303	1.471433344	1.476932286	1.378647399	1.316321417

shizy818 · 2025-01-17T09:53:23Z

Aligned Series:

TVList Sort Threshold	0	100	200	500	1000	2000	5000	10000	master
INGESTION	1585694.11	1976424.22	2003598.88	2067901.94	2034466.94	2004544.07	1938070.54	1841351.19	1734845.89
PRECISE_POINT	0	0	0	0	0	0	0	0	0
TIME_RANGE	1228.43	1480.37	1493.53	1523.19	1508.56	1505.3	1483.8	1425.03	1163.35
VALUE_RANGE	1233.58	1505.41	1516.68	1555.96	1536.99	1534.92	1503.98	1440.16	1168.66
AGG_RANGE	63.58	79.25	80.34	82.91	81.57	80.37	77.71	73.83	69.56
AGG_VALUE	62.65	78.09	79.16	81.7	80.38	79.2	76.57	72.75	68.54
AGG_RANGE_VALUE	62.19	77.52	78.58	81.1	79.79	78.62	76.01	72.22	68.04
GROUP_BY	835.24	1041.05	1055.36	1089.23	1071.62	1055.86	1020.85	969.9	913.8
LATEST_POINT	0	0	0	0	0	0	0	0	0
RANGE_QUERY_DESC	1220.13	1484.94	1492.06	1521.5	1503.96	1517.33	1484.09	1416.81	1151.73
VALUE_RANGE_QUERY_DESC	1221.62	1481.65	1496.19	1542.3	1515.41	1516.17	1484.18	1420.98	1166.05
GROUP_BY_DESC	817	1018.31	1032.32	1065.45	1048.22	1032.8	998.55	948.72	893.85
Comparison	0.914023606	1.139240365	1.154914135	1.19197852	1.172702355	1.155451138	1.117133747	1.061386139

shizy818 · 2025-01-17T09:58:18Z

8f78b21

When I use minimal heap to merge sort, it performs much better when there are a number of sorted tvlist ( tvlist_sort_threshold = 100). However it performs worse when there is only one tvlist (tvlist_sort_threshold = 0).

Not sure if I should revert this change.

HTHou · 2025-01-22T04:14:07Z

iotdb-core/node-commons/src/assembly/resources/conf/iotdb-system.properties.template

+# Datatype: int
+tvlist_sort_threshold=0
+
+# When the average point number of timeseries in memtable exceeds this, the memtable is flushed to disk. The default threshold is 100000.


Remove this line?

* Split non_aligned charge text chunk * dev non_aligned * dev aligned chunk split * new type * dev aligned binary chunk split * Fix binary size calculatation * fix IT * update IoTDBDuplicateTimeIT.java * fix pipe IT * change method names * add ut * add UT * remove useless methods * fix UT * fix /FileReaderManagerTest * fix win UT * add binary test * Add Aligned UTs * fix win ut * improve coverage * fix comments * fix windows UT * fix review * fix review * fix review * target chunk size count non binary * fix compile * fix UT * Tvlist feat new (#14616) * null bitmap for int tvlist * update min/max timestamp and sequential part of tvlist during insert * mutable & immutable tvlists in writable memchunk * copy-on-write array list * review comments part 1 * fix unit test errors * review comments part 2 * push down global time filter * fix MemPageReaderTest case * fix memory page offsets error * synchronized sort & MergeSortTvListIterator bug * tvlist_sort_threshold config property * bug fix: * out of mempage bounds check * overlapped data error during query * optimize TVListIterator & MergeSortTvListIterator * retrofit encode when tvlist_sort_threshold is zero * delay sort & statistic generation to query execution * fix: skip deleted data during encode * aligned time series part * fix: MemAlignedChunkReader page offset * performance issue: * change some list to array * remember row count in tvlist iterator * fix: memory chunk reader may read more points than expected in one page * update chunk & page statistic for aligend memchunk by column * revert: getAlignedValueForQuery * fix: * CopyOnWriteArrayList for AlignedTVList bitmaps * memory control of column access * refactor: Tim/Quick/Backward TVList * refactor: synchronized tvlist method: sort, putXXX * refactor: change list to array in AlignedTVList iterator * revert: remove CopyOnWriteArrayList * refactor: clone MergeSort iterator from ReadOnlyChunk * fix: clone working tvlist during flush if there is query on it * fix: writable mem chunk flush conditions * refactor: add annotation and variable/function rename * fix: * remove delete method in BinaryTVList * filter deleted data in WritableMemChunk encode * fix: remove getSortedTvListForQuery in SeriesRegionScan * fix: TsFileProcessorTest unit test * fix: IoTDBNullIdQueryIT.noMeasurementColumnsSelectTest * fix: delete column of aligned time series * fix: aligned timeseries encode bug * fix: IoTDBGroupByNaturalMonthIT * remove avgSeriesPointNumberThreshold setting * fix: IoTDBDeleteAlignedTimeseriesIT & AlignedTVListTest * fix: Copy globalTimeFilter due to GroupByMonthFilter * reset tmpLength for backward sort * * fix TVList clear * bitmap mark * sequence row count * hot-load TVLIST_SORT_THRESHOLD * fix: isNullValue caller * fix unit test * refactor: abstract prepareTvListMapForQuery method * refactor: clear/clone/expand indices and bitmap * merge sort using min heap * fix: WritableMemChunk deserialize * feat: add index mem cost for TVList * fix: hot-load tvlist_sort_threshold setting * remove needless line in property template --------- Co-authored-by: shizy <shizy04@gmail.com>

shizy818 added 30 commits December 5, 2024 09:46

null bitmap for int tvlist

ef5de53

update min/max timestamp and sequential part of tvlist during insert

7a15073

mutable & immutable tvlists in writable memchunk

c28fe3f

copy-on-write array list

6df2e10

review comments part 1

435d5da

fix unit test errors

d5fad22

review comments part 2

402c869

push down global time filter

5aaa59b

fix MemPageReaderTest case

cfc9e74

fix memory page offsets error

cd3f7d9

synchronized sort & MergeSortTvListIterator bug

e3d2d5e

tvlist_sort_threshold config property

73e9f83

bug fix:

0545ff1

* out of mempage bounds check * overlapped data error during query

optimize TVListIterator & MergeSortTvListIterator

2fcdcdb

retrofit encode when tvlist_sort_threshold is zero

45db409

delay sort & statistic generation to query execution

9411fea

fix: skip deleted data during encode

e3346d3

aligned time series part

8e0f8e5

fix: MemAlignedChunkReader page offset

6eedcf6

performance issue:

0a9c4ed

* change some list to array * remember row count in tvlist iterator

fix: memory chunk reader may read more points than expected in one page

9b0a6f8

update chunk & page statistic for aligend memchunk by column

23bf1f7

revert: getAlignedValueForQuery

298e16c

fix: * CopyOnWriteArrayList for AlignedTVList bitmaps

359929d

* memory control of column access

refactor: Tim/Quick/Backward TVList

6817370

refactor: synchronized tvlist method: sort, putXXX

a8beea5

refactor: change list to array in AlignedTVList iterator

6369293

revert: remove CopyOnWriteArrayList

a80bcc0

refactor: clone MergeSort iterator from ReadOnlyChunk

3e0b904

fix: clone working tvlist during flush if there is query on it

ca08605

reset tmpLength for backward sort

edf7952

jt2594838 approved these changes Jan 9, 2025

View reviewed changes

shizy818 commented Jan 10, 2025

View reviewed changes

...de/src/main/java/org/apache/iotdb/db/utils/datastructure/MergeSortAlignedTVListIterator.java Outdated Show resolved Hide resolved

shizy818 added 8 commits January 12, 2025 22:58

* fix TVList clear

d6bf701

* bitmap mark * sequence row count

hot-load TVLIST_SORT_THRESHOLD

89ad4d5

fix: isNullValue caller

a3feeee

fix unit test

cb7e13a

refactor: abstract prepareTvListMapForQuery method

1dc4192

refactor: clear/clone/expand indices and bitmap

bab27fd

merge sort using min heap

8f78b21

fix: WritableMemChunk deserialize

11f376c

shizy818 force-pushed the tvlist-feat-new branch from 68ce5ac to 11f376c Compare January 13, 2025 08:23

feat: add index mem cost for TVList

736e596

jt2594838 approved these changes Jan 14, 2025

View reviewed changes

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBDescriptor.java Outdated Show resolved Hide resolved

jt2594838 approved these changes Jan 14, 2025

View reviewed changes

fix: hot-load tvlist_sort_threshold setting

7fe1f85

jt2594838 approved these changes Jan 14, 2025

View reviewed changes

HTHou reviewed Jan 22, 2025

View reviewed changes

shizy818 added 2 commits January 22, 2025 15:39

merge force_ci/split_chunk

b09904e

remove needless line in property template

0900011

jt2594838 approved these changes Feb 7, 2025

View reviewed changes

jt2594838 merged commit e8e449e into apache:force_ci/split_chunk Feb 7, 2025
31 of 33 checks passed

shizy818 deleted the tvlist-feat-new branch March 20, 2025 10:38

shizy818 restored the tvlist-feat-new branch March 20, 2025 10:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tvlist feat new#14616

Tvlist feat new#14616
jt2594838 merged 56 commits intoapache:force_ci/split_chunkfrom
shizy818:tvlist-feat-new

shizy818 commented Jan 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shizy818 commented Jan 17, 2025

Uh oh!

shizy818 commented Jan 17, 2025 •

edited

Loading

Uh oh!

shizy818 commented Jan 17, 2025

Uh oh!

shizy818 commented Jan 17, 2025

Uh oh!

HTHou Jan 22, 2025

Uh oh!

shizy818 Jan 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

shizy818 commented Jan 2, 2025

Description

Content1 ...

Content2 ...

Content3 ...

Key changed/added classes (or packages if there are too many classes) in this PR

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shizy818 commented Jan 17, 2025

Uh oh!

shizy818 commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shizy818 commented Jan 17, 2025

Uh oh!

shizy818 commented Jan 17, 2025

Uh oh!

HTHou Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

shizy818 Jan 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

shizy818 commented Jan 17, 2025 •

edited

Loading