Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HUDI-5487] Reduce duplicate Logs in ExternalSpillableMap #7579

Merged
merged 1 commit into from Jan 31, 2023

Conversation

cxzl25
Copy link
Contributor

@cxzl25 cxzl25 commented Dec 29, 2022

Change Logs

Reduce duplicate Logs in ExternalSpillableMap

We see hundreds of thousands of duplicate logs in the executor log.

22/12/26 21:13:40,864 [Executor task launch worker for task 0.0 in stage 480.0 (TID 211376)] INFO ExternalSpillableMap: Update Estimated Payload size to => 4567
22/12/26 21:13:40,864 [Executor task launch worker for task 0.0 in stage 480.0 (TID 211376)] INFO ExternalSpillableMap: Update Estimated Payload size to => 4567
22/12/26 21:13:40,864 [Executor task launch worker for task 0.0 in stage 480.0 (TID 211376)] INFO ExternalSpillableMap: Update Estimated Payload size to => 4567
22/12/26 21:13:40,864 [Executor task launch worker for task 0.0 in stage 480.0 (TID 211376)] INFO ExternalSpillableMap: Update Estimated Payload size to => 4567
22/12/26 21:13:40,864 [Executor task launch worker for task 0.0 in stage 480.0 (TID 211376)] INFO ExternalSpillableMap: Update Estimated Payload size to => 4567
22/12/26 21:13:40,864 [Executor task launch worker for task 0.0 in stage 480.0 (TID 211376)] INFO ExternalSpillableMap: Update Estimated Payload size to => 4567
22/12/26 21:13:40,864 [Executor task launch worker for task 0.0 in stage 480.0 (TID 211376)] INFO ExternalSpillableMap: Update Estimated Payload size to => 4567

Impact

Reduce noisy logs and reduce log size

Risk level (write none, low medium or high below)

none

Documentation Update

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@cxzl25
Copy link
Contributor Author

cxzl25 commented Dec 29, 2022

@hudi-bot run azure

@hudi-bot
Copy link

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@cxzl25
Copy link
Contributor Author

cxzl25 commented Jan 30, 2023

Help review. Thanks.
@XuQianJin-Stars @alexeykudinkin

@danny0405 danny0405 merged commit 1377143 into apache:master Jan 31, 2023
fengjian428 pushed a commit to fengjian428/hudi that referenced this pull request Jan 31, 2023
fengjian428 pushed a commit to fengjian428/hudi that referenced this pull request Apr 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

4 participants