Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Refactor] Synchronize OLAP external table metadata when loading data(cherry-pick 3.1) #31651

Merged
merged 3 commits into from
Sep 25, 2023
Merged

[Refactor] Synchronize OLAP external table metadata when loading data(cherry-pick 3.1) #31651

merged 3 commits into from
Sep 25, 2023

Conversation

gengjun-git
Copy link
Contributor

backport #24739 #30124

@gengjun-git
[Refactor] Synchronize OLAP external table metadata when loading data (…
09f5313 
…StarRocks#24739)

Currently, StarRocks use StarRocksRepository to synchronize OLAP
external table metadata every 10 seconds, which is
inefficient, because the meta data will never be used when there is no
load job on that external table. Change the synchronization to be
triggered when there is load job on that table.

Since the synchronization is time costly, so we should release the
database lock when synchronizing meta data.

Signed-off-by: gengjun-git <gengjun@starrocks.com>
@gengjun-git
[BugFix] Fix OLAP external table sync bug if the target table is part…
f75617b 
…itioned (StarRocks#30124)

Signed-off-by: gengjun-git <gengjun@starrocks.com>
@gengjun-git
fix
9b2899c 
Signed-off-by: gengjun-git <gengjun@starrocks.com>
@sonarcloud
Copy link

sonarcloud bot commented Sep 22, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 9 Code Smells

0.0% 0.0% Coverage
0.0% 0.0% Duplication

warning The version of Java (11.0.20.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more here

@wanpengfei-git
Copy link
Collaborator

[FE Incremental Coverage Report]

😞 fail : 142 / 192 (73.96%)

file detail

path covered_line new_line coverage not_covered_line_detail
🔵 com/starrocks/catalog/Database.java 0 1 00.00% [244]
🔵 com/starrocks/external/starrocks/TableMetaSyncer.java 0 2 00.00% [54, 60]
🔵 com/starrocks/sql/analyzer/InsertAnalyzer.java 1 12 08.33% [86, 322, 323, 324, 325, 326, 327, 330, 332, 333, 336]
🔵 com/starrocks/catalog/ExternalOlapTable.java 134 170 78.82% [350, 351, 352, 353, 400, 401, 402, 403, 404, 406, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467, 468, 469, 471, 472, 562, 576, 577, 578, 579, 580, 581, 582, 583, 584]
🔵 com/starrocks/sql/analyzer/QueryAnalyzer.java 2 2 100.00% []
🔵 com/starrocks/sql/analyzer/AnalyzerUtils.java 2 2 100.00% []
🔵 com/starrocks/qe/StmtExecutor.java 3 3 100.00% []

@wanpengfei-git
Copy link
Collaborator

[BE Incremental Coverage Report]

😍 pass : 0 / 0 (0%)

@gengjun-git gengjun-git merged commit 587f4ce into StarRocks:branch-3.1 Sep 25, 2023
24 of 25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Youngwb Youngwb Youngwb approved these changes

Assignees

@gengjun-git gengjun-git

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@gengjun-git @wanpengfei-git @Youngwb