[Refactor] Synchronize OLAP external table metadata when loading data #24739

gengjun-git · 2023-06-06T06:58:01Z

Currently, StarRocks use StarRocksRepository to synchronize OLAP external table metadata every 10 seconds, which is
inefficient, because the meta data will never be used when there is no load job on that external table. Change the synchronization to be triggered when there is load job on that table.

Since the synchronization is time costly, so we should release the database lock when synchronizing meta data.

What type of PR is this:

Checklist:

I have added test cases for my bug fix or my new feature
This pr will affect users' behaviors
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

Signed-off-by: gengjun-git <gengjun@starrocks.com>

fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java

Signed-off-by: gengjun-git <gengjun@starrocks.com>

fe/fe-core/src/main/java/com/starrocks/sql/analyzer/InsertAnalyzer.java

Signed-off-by: gengjun-git <gengjun@starrocks.com>

sonarcloud · 2023-06-07T12:36:51Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
6 Code Smells

0.0% Coverage
0.0% Duplication

wanpengfei-git · 2023-06-07T12:51:21Z

[FE PR Coverage Check]

😞 fail : 6 / 186 (03.23%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/catalog/Database.java	0	1	00.00%	[244]
🔵	com/starrocks/external/starrocks/TableMetaSyncer.java	0	2	00.00%	[54, 60]
🔵	com/starrocks/sql/analyzer/QueryAnalyzer.java	0	1	00.00%	[923]
🔵	com/starrocks/catalog/ExternalOlapTable.java	0	165	00.00%	[339, 340, 341, 342, 360, 362, 363, 364, 365, 367, 368, 369, 370, 371, 372, 373, 374, 376, 377, 378, 379, 380, 381, 382, 384, 387, 388, 389, 392, 393, 394, 396, 411, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425, 426, 427, 429, 430, 431, 432, 434, 435, 436, 437, 438, 439, 440, 441, 442, 443, 444, 445, 446, 447, 449, 450, 452, 454, 455, 457, 458, 459, 460, 461, 462, 463, 465, 466, 468, 469, 471, 472, 474, 476, 477, 478, 479, 480, 481, 482, 484, 487, 488, 490, 491, 493, 494, 496, 497, 498, 499, 500, 501, 502, 505, 506, 507, 510, 511, 512, 513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523, 524, 525, 526, 527, 528, 530, 531, 533, 534, 535, 536, 537, 538, 539, 540, 542, 546, 547, 548, 550, 551, 552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 564, 565, 566, 567, 568, 569, 573, 576, 577, 578]
🔵	com/starrocks/sql/analyzer/InsertAnalyzer.java	1	12	08.33%	[84, 294, 295, 296, 297, 298, 299, 302, 304, 305, 308]
🔵	com/starrocks/sql/analyzer/AnalyzerUtils.java	2	2	100.00%	[]
🔵	com/starrocks/qe/StmtExecutor.java	3	3	100.00%	[]

gengjun-git · 2023-06-12T12:48:25Z

@Mergifyio backport branch-3.1 branch-3.0 branch-2.5 branch-2.4 branch-2.3

mergify · 2023-06-12T12:48:32Z

backport branch-3.1 branch-3.0 branch-2.5 branch-2.4 branch-2.3

✅ Backports have been created

#25087 [Refactor] Synchronize OLAP external table metadata when loading data (backport #24739) has been created for branch branch-3.1
#25088 [Refactor] Synchronize OLAP external table metadata when loading data (backport #24739) has been created for branch branch-3.0 but encountered conflicts
#25089 [Refactor] Synchronize OLAP external table metadata when loading data (backport #24739) has been created for branch branch-2.5 but encountered conflicts
#25090 [Refactor] Synchronize OLAP external table metadata when loading data (backport #24739) has been created for branch branch-2.4 but encountered conflicts
#25091 [Refactor] Synchronize OLAP external table metadata when loading data (backport #24739) has been created for branch branch-2.3 but encountered conflicts

…#24739) Currently, StarRocks use StarRocksRepository to synchronize OLAP external table metadata every 10 seconds, which is inefficient, because the meta data will never be used when there is no load job on that external table. Change the synchronization to be triggered when there is load job on that table. Since the synchronization is time costly, so we should release the database lock when synchronizing meta data. Signed-off-by: gengjun-git <gengjun@starrocks.com> (cherry picked from commit ef1fc6a)

…#24739) Currently, StarRocks use StarRocksRepository to synchronize OLAP external table metadata every 10 seconds, which is inefficient, because the meta data will never be used when there is no load job on that external table. Change the synchronization to be triggered when there is load job on that table. Since the synchronization is time costly, so we should release the database lock when synchronizing meta data. Signed-off-by: gengjun-git <gengjun@starrocks.com> (cherry picked from commit ef1fc6a) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/catalog/ExternalOlapTable.java # fe/fe-core/src/main/java/com/starrocks/catalog/OlapTable.java # fe/fe-core/src/main/java/com/starrocks/external/starrocks/StarRocksRepository.java # fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java # fe/fe-core/src/main/java/com/starrocks/server/GlobalStateMgr.java # fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java # fe/fe-core/src/main/java/com/starrocks/sql/analyzer/AnalyzerUtils.java # fe/fe-core/src/main/java/com/starrocks/sql/analyzer/InsertAnalyzer.java # fe/fe-core/src/main/java/com/starrocks/sql/analyzer/QueryAnalyzer.java

…#24739) Currently, StarRocks use StarRocksRepository to synchronize OLAP external table metadata every 10 seconds, which is inefficient, because the meta data will never be used when there is no load job on that external table. Change the synchronization to be triggered when there is load job on that table. Since the synchronization is time costly, so we should release the database lock when synchronizing meta data. Signed-off-by: gengjun-git <gengjun@starrocks.com> (cherry picked from commit ef1fc6a) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/catalog/ExternalOlapTable.java # fe/fe-core/src/main/java/com/starrocks/external/starrocks/StarRocksRepository.java # fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java # fe/fe-core/src/main/java/com/starrocks/server/GlobalStateMgr.java # fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java # fe/fe-core/src/main/java/com/starrocks/sql/analyzer/InsertAnalyzer.java # fe/fe-core/src/main/java/com/starrocks/sql/analyzer/QueryAnalyzer.java

…#24739) Currently, StarRocks use StarRocksRepository to synchronize OLAP external table metadata every 10 seconds, which is inefficient, because the meta data will never be used when there is no load job on that external table. Change the synchronization to be triggered when there is load job on that table. Since the synchronization is time costly, so we should release the database lock when synchronizing meta data. Signed-off-by: gengjun-git <gengjun@starrocks.com> (cherry picked from commit ef1fc6a) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/catalog/ExternalOlapTable.java # fe/fe-core/src/main/java/com/starrocks/catalog/OlapTable.java # fe/fe-core/src/main/java/com/starrocks/external/starrocks/StarRocksRepository.java # fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java # fe/fe-core/src/main/java/com/starrocks/server/GlobalStateMgr.java # fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java # fe/fe-core/src/main/java/com/starrocks/sql/analyzer/AnalyzerUtils.java # fe/fe-core/src/main/java/com/starrocks/sql/analyzer/InsertAnalyzer.java # fe/fe-core/src/main/java/com/starrocks/sql/analyzer/QueryAnalyzer.java

…StarRocks#24739) Currently, StarRocks use StarRocksRepository to synchronize OLAP external table metadata every 10 seconds, which is inefficient, because the meta data will never be used when there is no load job on that external table. Change the synchronization to be triggered when there is load job on that table. Since the synchronization is time costly, so we should release the database lock when synchronizing meta data. Signed-off-by: gengjun-git <gengjun@starrocks.com>

…#24739) Currently, StarRocks use StarRocksRepository to synchronize OLAP external table metadata every 10 seconds, which is inefficient, because the meta data will never be used when there is no load job on that external table. Change the synchronization to be triggered when there is load job on that table. Since the synchronization is time costly, so we should release the database lock when synchronizing meta data. Signed-off-by: gengjun-git <gengjun@starrocks.com>

… loading data (#25496) Currently, StarRocks use StarRocksRepository to synchronize OLAP external table metadata every 10 seconds, which is inefficient, because the meta data will never be used when there is no load job on that external table. Change the synchronization to be triggered when there is load job on that table. Since the synchronization is time costly, so we should release the database lock when synchronizing meta data. backport #24739 Signed-off-by: gengjun-git <gengjun@starrocks.com>

…#24739) Currently, StarRocks use StarRocksRepository to synchronize OLAP external table metadata every 10 seconds, which is inefficient, because the meta data will never be used when there is no load job on that external table. Change the synchronization to be triggered when there is load job on that table. Since the synchronization is time costly, so we should release the database lock when synchronizing meta data. Signed-off-by: gengjun-git <gengjun@starrocks.com>

…StarRocks#24739) Currently, StarRocks use StarRocksRepository to synchronize OLAP external table metadata every 10 seconds, which is inefficient, because the meta data will never be used when there is no load job on that external table. Change the synchronization to be triggered when there is load job on that table. Since the synchronization is time costly, so we should release the database lock when synchronizing meta data. Signed-off-by: gengjun-git <gengjun@starrocks.com>

…(cherry-pick 3.1) (#31651) backport #24739 #30124 Signed-off-by: gengjun-git <gengjun@starrocks.com>

fix

90b2751

Signed-off-by: gengjun-git <gengjun@starrocks.com>

mergify bot assigned gengjun-git Jun 6, 2023

nshangyiming reviewed Jun 6, 2023

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java Show resolved Hide resolved

fix compile error

3ba32b2

Signed-off-by: gengjun-git <gengjun@starrocks.com>

nshangyiming reviewed Jun 6, 2023

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/sql/analyzer/InsertAnalyzer.java Show resolved Hide resolved

fe/fe-core/src/main/java/com/starrocks/sql/analyzer/InsertAnalyzer.java Show resolved Hide resolved

fix

0a662dc

Signed-off-by: gengjun-git <gengjun@starrocks.com>

nshangyiming approved these changes Jun 8, 2023

View reviewed changes

gengjun-git enabled auto-merge (squash) June 12, 2023 06:32

Astralidea approved these changes Jun 12, 2023

View reviewed changes

gengjun-git merged commit ef1fc6a into StarRocks:main Jun 12, 2023
26 of 27 checks passed

gengjun-git mentioned this pull request Jun 16, 2023

[Refactor][Cherry-Pick] Synchronize OLAP external table metadata when loading data #25496

Merged

gengjun-git mentioned this pull request Jun 16, 2023

[Refactor][Cherry-Pick] Synchronize OLAP external table metadata when loading data #25499

Merged

gengjun-git mentioned this pull request Jun 16, 2023

[Refactor][Cherry-Pick] Synchronize OLAP external table metadata when loading data #25500

Merged

gengjun-git mentioned this pull request Sep 22, 2023

[Refactor] Synchronize OLAP external table metadata when loading data(cherry-pick 3.1) #31651

Merged

gengjun-git added a commit that referenced this pull request Sep 25, 2023

[Refactor] Synchronize OLAP external table metadata when loading data…

587f4ce

…(cherry-pick 3.1) (#31651) backport #24739 #30124 Signed-off-by: gengjun-git <gengjun@starrocks.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Refactor] Synchronize OLAP external table metadata when loading data #24739

[Refactor] Synchronize OLAP external table metadata when loading data #24739

gengjun-git commented Jun 6, 2023 •

edited

sonarcloud bot commented Jun 7, 2023

wanpengfei-git commented Jun 7, 2023

gengjun-git commented Jun 12, 2023

mergify bot commented Jun 12, 2023 •

edited

[Refactor] Synchronize OLAP external table metadata when loading data #24739

[Refactor] Synchronize OLAP external table metadata when loading data #24739

Conversation

gengjun-git commented Jun 6, 2023 • edited

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

sonarcloud bot commented Jun 7, 2023

wanpengfei-git commented Jun 7, 2023

[FE PR Coverage Check]

file detail

gengjun-git commented Jun 12, 2023

mergify bot commented Jun 12, 2023 • edited

✅ Backports have been created

gengjun-git commented Jun 6, 2023 •

edited

mergify bot commented Jun 12, 2023 •

edited