Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(logger): upgrade logforth to support global max_files limit for log_dir #17637

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

BohuTANG
Copy link
Member

@BohuTANG BohuTANG commented Mar 21, 2025

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

Upgrade logforth with Global File Limit Capability

Upgrades to logforth from https://github.com/datafuse-extras/logforth (global-max-files-v0.14 branch), which adds the ability to limit the total number of log files in log_dir via PR #1.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - tests already in datafuse-extras/logforth

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe): enhance the log files limitition

This change is Reviewable

@github-actions github-actions bot added the pr-chore this PR only has small changes that no need to record, like coding styles. label Mar 21, 2025
@BohuTANG BohuTANG changed the title chore: upgrade logforth to support max_files limit chore(logger): upgrade logforth to support max_files limit Mar 21, 2025
@BohuTANG BohuTANG requested a review from Xuanwo March 21, 2025 09:41
@BohuTANG BohuTANG marked this pull request as ready for review March 21, 2025 09:41
@Xuanwo
Copy link
Member

Xuanwo commented Mar 21, 2025

This is a temporary solution, right? I see logforth 0.23 has already supported the max file limit: https://github.com/fast/logforth/blob/d9620447ba9be53a730d8960f103445de79076a9/src/append/rolling_file/rolling.rs#L137-L142

@BohuTANG
Copy link
Member Author

BohuTANG commented Mar 21, 2025

This is a temporary solution, right? I see logforth 0.23 has already supported the max file limit: https://github.com/fast/logforth/blob/d9620447ba9be53a730d8960f103445de79076a9/src/append/rolling_file/rolling.rs#L137-L142

No, this is the final solution.
The max_files is for rotation(if hourly the max_files mean not over max_files in a hour) not global all in log_dir, see : fast/logforth#107
BTW, it's hard to use the latest version of logforth, there are many api need to change, so we don't have the plan to upgrade to the latest version.

@BohuTANG BohuTANG changed the title chore(logger): upgrade logforth to support max_files limit chore(logger): upgrade logforth to support global max_files limit for log_dir Mar 21, 2025
@wubx
Copy link
Member

wubx commented Mar 23, 2025

The log section with:

[log.file]
limit = 12
max-size=1024
format = "text"
include_node_id=true
dir = "/var/log/databend"
level = "WARN,databend=info,opendal=info,openraft=info"

Test use

for i in `seq 1 1000` ; do echo "select $i"|bendsql ; done

"It feels like the number of generated log files is far beyond what was expected."

image

@BohuTANG BohuTANG marked this pull request as draft March 24, 2025 08:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-chore this PR only has small changes that no need to record, like coding styles.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

chore: enable limit total number of log files
3 participants