Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Refactor] Synchronize OLAP external table metadata when loading data (backport #24739) #25088

Closed
wants to merge 1 commit into from

Conversation

mergify[bot]
Copy link
Contributor

@mergify mergify bot commented Jun 12, 2023

This is an automatic backport of pull request #24739 done by Mergify.
Cherry-pick of ef1fc6a has failed:

On branch mergify/bp/branch-3.0/pr-24739
Your branch is up to date with 'origin/branch-3.0'.

You are currently cherry-picking commit ef1fc6a40.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   fe/fe-core/src/main/java/com/starrocks/catalog/Database.java
	modified:   fe/fe-core/src/main/java/com/starrocks/catalog/OlapTable.java
	modified:   fe/fe-core/src/main/java/com/starrocks/external/starrocks/TableMetaSyncer.java
	modified:   fe/fe-core/src/main/java/com/starrocks/sql/analyzer/AnalyzerUtils.java

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	both modified:   fe/fe-core/src/main/java/com/starrocks/catalog/ExternalOlapTable.java
	deleted by them: fe/fe-core/src/main/java/com/starrocks/external/starrocks/StarRocksRepository.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/server/GlobalStateMgr.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/sql/analyzer/InsertAnalyzer.java
	both modified:   fe/fe-core/src/main/java/com/starrocks/sql/analyzer/QueryAnalyzer.java

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally


Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

  • @Mergifyio refresh will re-evaluate the rules
  • @Mergifyio rebase will rebase this PR on its base branch
  • @Mergifyio update will merge the base branch into this PR
  • @Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

  • look at your merge queues
  • generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.com

…#24739)

Currently, StarRocks use StarRocksRepository to synchronize OLAP
external table metadata every 10 seconds, which is
inefficient, because the meta data will never be used when there is no
load job on that external table. Change the synchronization to be
triggered when there is load job on that table.

Since the synchronization is time costly, so we should release the
database lock when synchronizing meta data.

Signed-off-by: gengjun-git <gengjun@starrocks.com>
(cherry picked from commit ef1fc6a)

# Conflicts:
#	fe/fe-core/src/main/java/com/starrocks/catalog/ExternalOlapTable.java
#	fe/fe-core/src/main/java/com/starrocks/external/starrocks/StarRocksRepository.java
#	fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java
#	fe/fe-core/src/main/java/com/starrocks/server/GlobalStateMgr.java
#	fe/fe-core/src/main/java/com/starrocks/server/LocalMetastore.java
#	fe/fe-core/src/main/java/com/starrocks/sql/analyzer/InsertAnalyzer.java
#	fe/fe-core/src/main/java/com/starrocks/sql/analyzer/QueryAnalyzer.java
@mergify mergify bot added the conflicts label Jun 12, 2023
@wanpengfei-git wanpengfei-git enabled auto-merge (rebase) June 12, 2023 12:49
@mergify mergify bot closed this Jun 12, 2023
auto-merge was automatically disabled June 12, 2023 12:49

Pull request was closed

@mergify
Copy link
Contributor Author

mergify bot commented Jun 12, 2023

@mergify[bot]: Backport conflict, please reslove the conflict and resubmit the pr

@mergify mergify bot deleted the mergify/bp/branch-3.0/pr-24739 branch June 12, 2023 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants