-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Iceberg metadata super optimization #43460
Open
1 of 9 tasks
Labels
Comments
24 tasks
This was referenced Apr 17, 2024
Merged
24 tasks
24 tasks
24 tasks
24 tasks
24 tasks
24 tasks
This was referenced Apr 24, 2024
24 tasks
24 tasks
This was referenced Apr 26, 2024
This was referenced May 6, 2024
Merged
Merged
imay
pushed a commit
that referenced
this issue
May 11, 2024
…uted_plan interface (#45404) Why I'm doing: we use ConnectContext.get() to get Session variable in the iceberg job planning. If we don't execute planning in the query thread, and can't get the connect context. There are two cases: In the prepareMetadata, the job planning is executed in the thread pool distributed plan mode, the metadata collect job is executed in the another thread. 1 && 2. plan_mode is distributed and running under the thread pool So we need to adapt to the two case with the connect context. What I'm doing: Adapting ConnectContext and Tracers under multi-threading Fixes #issue #43460 Signed-off-by: stephen <stephen5217@163.com>
stephen-shelby
added a commit
to stephen-shelby/starrocks
that referenced
this issue
May 15, 2024
…uted_plan interface (StarRocks#45404) Why I'm doing: we use ConnectContext.get() to get Session variable in the iceberg job planning. If we don't execute planning in the query thread, and can't get the connect context. There are two cases: In the prepareMetadata, the job planning is executed in the thread pool distributed plan mode, the metadata collect job is executed in the another thread. 1 && 2. plan_mode is distributed and running under the thread pool So we need to adapt to the two case with the connect context. What I'm doing: Adapting ConnectContext and Tracers under multi-threading Fixes #issue StarRocks#43460 Signed-off-by: stephen <stephen5217@163.com>
This was referenced May 15, 2024
24 tasks
node
pushed a commit
to vivo/starrocks
that referenced
this issue
May 17, 2024
…uted_plan interface (StarRocks#45404) Why I'm doing: we use ConnectContext.get() to get Session variable in the iceberg job planning. If we don't execute planning in the query thread, and can't get the connect context. There are two cases: In the prepareMetadata, the job planning is executed in the thread pool distributed plan mode, the metadata collect job is executed in the another thread. 1 && 2. plan_mode is distributed and running under the thread pool So we need to adapt to the two case with the connect context. What I'm doing: Adapting ConnectContext and Tracers under multi-threading Fixes #issue StarRocks#43460 Signed-off-by: stephen <stephen5217@163.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Enhancement
This issue is used to trace iceberg metadata related optimization patches. The first version patch is #43459
There are currently four important optimizations:
There is still a lot of optimization work to come, some of which are as follows:
as of
.RemoteFileInfo
The text was updated successfully, but these errors were encountered: