[Improve] Speed up ck file attach (#6091)#6138
[Improve] Speed up ck file attach (#6091)#6138zhanggougou wants to merge 1 commit intoapache:devfrom
Conversation
|
cc @Hisoka-X |
|
@Hisoka-X |
|
You need to explain the reason for the change and add test cases |
|
@hailin0 I think all corresponding clickhousefile files should be generated before attaching, otherwise some files may fail to be generated, resulting in failure to attach some files. We should not attach a clickhousefile file immediately after it is generated. |
+1 |
|
This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 7 days if no further activity occurs. |
|
This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request. |
Purpose of this pull request
improve the speed of transfer data from hive to ck
Does this PR introduce any user-facing change?
no
Tips:because attach data is in executor,it means the data will keep increasing when task is running.
if your task has failed ,you ck will also have data. so you should handle data before run task to keep idempotent【like dataX preSql】
How was this patch tested?
after this imporve and next ”chown improve“,300KW records from hive to ck,cost minutes reduced from 100+ to 60
Check list
New License Guide
release-note.