Skip to content

[Improve] Speed up ck file attach (#6091)#6138

Closed
zhanggougou wants to merge 1 commit intoapache:devfrom
zhanggougou:feature/speed_ckFile_attach
Closed

[Improve] Speed up ck file attach (#6091)#6138
zhanggougou wants to merge 1 commit intoapache:devfrom
zhanggougou:feature/speed_ckFile_attach

Conversation

@zhanggougou
Copy link

Purpose of this pull request

improve the speed of transfer data from hive to ck

Does this PR introduce any user-facing change?

no
Tips:because attach data is in executor,it means the data will keep increasing when task is running.
if your task has failed ,you ck will also have data. so you should handle data before run task to keep idempotent【like dataX preSql】

How was this patch tested?

after this imporve and next ”chown improve“,300KW records from hive to ck,cost minutes reduced from 100+ to 60

Check list

@zhanggougou zhanggougou closed this Jan 5, 2024
@zhanggougou zhanggougou reopened this Jan 5, 2024
@hailin0
Copy link
Member

hailin0 commented Jan 15, 2024

cc @Hisoka-X

@hailin0
Copy link
Member

hailin0 commented Feb 18, 2024

300KW records from hive to ck,cost minutes reduced from 100+ to 60

@Hisoka-X
sounds good

@hailin0
Copy link
Member

hailin0 commented Mar 12, 2024

You need to explain the reason for the change and add test cases

@caicancai
Copy link
Member

caicancai commented Dec 15, 2024

@hailin0 I think all corresponding clickhousefile files should be generated before attaching, otherwise some files may fail to be generated, resulting in failure to attach some files.

We should not attach a clickhousefile file immediately after it is generated.

@Hisoka-X
Copy link
Member

We should not attach a clickhousefile file immediately after it is generated.

+1

@github-actions
Copy link

This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 7 days if no further activity occurs.

@github-actions github-actions bot added the stale label Jul 13, 2025
@github-actions
Copy link

This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.

@github-actions github-actions bot closed this Jul 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants