max_log_files_feature #2323

ozgunozerk · 2022-09-24T20:34:08Z

Motivation

tracing-appender does not have Rotation based on size yet.
Also, it doesn't have the feature of keeping the most recent N log files

I believe the second feature is more easy to implement, and also will partially solve the Rotation based on size problem. Because people may choose hourly or daily rotation based on their needs, and put an extra boundary of keep the last 5 files for example. Of course it won't handle all the edge cases for Rotation based on size. But it will cover most of the scenarios. And also, it is a good feature to have on its own :)

Solution

Introduce another field called keep_last: Option<usize> to the Inner of RollingFileAppender struct.
I managed to did not touch any of the existing functions, so it WON'T BE A BREAKING CHANGE. Yay :)

The solution is, whenever the rotation should happen, the refresh_writer() is called. So I embed the following logic into that function:

1- check the log folder and detect the log files
2- if there are more log files than the last_keep amount
3- store the filenames in a vector, and sort them by their dates (dates are already present in the filename)
4- keep deleting the oldest ones, till we have desired amount of log files in the log folder

P.S. this PR was opened before, but got closed since it would be easier for the maintainers to target master branch instead of v0.1.x
Also, @CBenoit contributed to this PR, it would be great to give credit to him :)

hawkw

I previously approved this on the original PR against the v0.1.x branch, so this looks good to me, provided CI passes! I'll make sure to give credit to @CBenoit when merging.

Thanks again for the PR!

tracing-appender/src/rolling/builder.rs

hawkw

I noticed one last thing that I think was missed while porting this change over from your branch against v0.1.x. It looks like this branch is missing the change to add the "parsing" feature flag to the dependency on time, so this won't build when used as a dependency unless user code is already enabling time's "parsing" feature. This isn't an issue for the tests, because they have a dev-dependency that already enables that feature:

tracing/tracing-appender/Cargo.toml

Line 38 in fb7cb4a

    
           time = { version = "0.3.2", default-features = false, features = ["formatting", "parsing"] }

I think before we can merge this, we have to change this line:

tracing/tracing-appender/Cargo.toml

Line 25 in fb7cb4a

    
           time = { version = "0.3.2", default-features = false, features = ["formatting"] }

so that it also enables the "parsing" feature flag. Then, the dev dependency can just be removed.

Once that's addressed, this will be good to go!

hawkw · 2022-09-24T20:43:09Z

oh, i guess CI actually is failing due to the missing feature flag: https://github.com/tokio-rs/tracing/actions/runs/3119699075/jobs/5059829752#step:7:79

ozgunozerk · 2022-09-24T20:44:13Z

your CI's are fast, I'm impressed :)

hawkw

looks good to me as soon as CI passes (which i think it ought to now!)

@CBenoit

## Motivation `tracing-appender` does not have `Rotation` based on size yet. Also, it doesn't have the feature of keeping the most recent `N` log files I believe the second feature is more easy to implement, and also will partially solve the `Rotation` based on size problem. Because people may choose `hourly` or `daily` rotation based on their needs, and put an extra boundary of `keep the last 5 files` for example. Of course it won't handle all the edge cases for `Rotation` based on size. But it will cover most of the scenarios. And also, it is a good feature to have on its own :) ## Solution Introduce another field called `max_files: Option<usize>` to the `Inner` of `RollingFileAppender` struct. I managed to did not touch any of the existing functions, so it **WON'T BE A BREAKING CHANGE**. Yay :) The solution is, whenever the rotation should happen, the `refresh_writer()` is called. So I embed the following logic into that function: 1- check the log folder and detect the log files 2- if there are more log files than the `max_files` amount 3- store the filenames in a vector, and sort them by their dates (dates are already present in the filename) 4- keep deleting the oldest ones, till we have desired amount of log files in the log folder P.S. this PR was opened before, but got closed since it would be easier for the maintainers to target `master` branch instead of `v0.1.x` Also, @CBenoit contributed to this PR, it would be great to give credit to him :) Co-authored-by: Benoît Cortier <bcortier@proton.me> Co-authored-by: Eliza Weisman <eliza@buoyant.io>

@CBenoit

## Motivation `tracing-appender` does not have `Rotation` based on size yet. Also, it doesn't have the feature of keeping the most recent `N` log files I believe the second feature is more easy to implement, and also will partially solve the `Rotation` based on size problem. Because people may choose `hourly` or `daily` rotation based on their needs, and put an extra boundary of `keep the last 5 files` for example. Of course it won't handle all the edge cases for `Rotation` based on size. But it will cover most of the scenarios. And also, it is a good feature to have on its own :) ## Solution Introduce another field called `max_files: Option<usize>` to the `Inner` of `RollingFileAppender` struct. I managed to did not touch any of the existing functions, so it **WON'T BE A BREAKING CHANGE**. Yay :) The solution is, whenever the rotation should happen, the `refresh_writer()` is called. So I embed the following logic into that function: 1- check the log folder and detect the log files 2- if there are more log files than the `max_files` amount 3- store the filenames in a vector, and sort them by their dates (dates are already present in the filename) 4- keep deleting the oldest ones, till we have desired amount of log files in the log folder P.S. this PR was opened before, but got closed since it would be easier for the maintainers to target `master` branch instead of `v0.1.x` Also, @CBenoit contributed to this PR, it would be great to give credit to him :) Co-authored-by: Benoît Cortier <bcortier@proton.me> Co-authored-by: Eliza Weisman <eliza@buoyant.io>

@CBenoit

## Motivation `tracing-appender` does not have `Rotation` based on size yet. Also, it doesn't have the feature of keeping the most recent `N` log files I believe the second feature is more easy to implement, and also will partially solve the `Rotation` based on size problem. Because people may choose `hourly` or `daily` rotation based on their needs, and put an extra boundary of `keep the last 5 files` for example. Of course it won't handle all the edge cases for `Rotation` based on size. But it will cover most of the scenarios. And also, it is a good feature to have on its own :) ## Solution Introduce another field called `max_files: Option<usize>` to the `Inner` of `RollingFileAppender` struct. I managed to did not touch any of the existing functions, so it **WON'T BE A BREAKING CHANGE**. Yay :) The solution is, whenever the rotation should happen, the `refresh_writer()` is called. So I embed the following logic into that function: 1- check the log folder and detect the log files 2- if there are more log files than the `max_files` amount 3- store the filenames in a vector, and sort them by their dates (dates are already present in the filename) 4- keep deleting the oldest ones, till we have desired amount of log files in the log folder P.S. this PR was opened before, but got closed since it would be easier for the maintainers to target `master` branch instead of `v0.1.x` Also, @CBenoit contributed to this PR, it would be great to give credit to him :) Co-authored-by: Benoît Cortier <bcortier@proton.me> Co-authored-by: Eliza Weisman <eliza@buoyant.io>

@CBenoit

## Motivation `tracing-appender` does not have `Rotation` based on size yet. Also, it doesn't have the feature of keeping the most recent `N` log files I believe the second feature is more easy to implement, and also will partially solve the `Rotation` based on size problem. Because people may choose `hourly` or `daily` rotation based on their needs, and put an extra boundary of `keep the last 5 files` for example. Of course it won't handle all the edge cases for `Rotation` based on size. But it will cover most of the scenarios. And also, it is a good feature to have on its own :) ## Solution Introduce another field called `max_files: Option<usize>` to the `Inner` of `RollingFileAppender` struct. I managed to did not touch any of the existing functions, so it **WON'T BE A BREAKING CHANGE**. Yay :) The solution is, whenever the rotation should happen, the `refresh_writer()` is called. So I embed the following logic into that function: 1- check the log folder and detect the log files 2- if there are more log files than the `max_files` amount 3- store the filenames in a vector, and sort them by their dates (dates are already present in the filename) 4- keep deleting the oldest ones, till we have desired amount of log files in the log folder P.S. this PR was opened before, but got closed since it would be easier for the maintainers to target `master` branch instead of `v0.1.x` Also, @CBenoit contributed to this PR, it would be great to give credit to him :) Co-authored-by: Benoît Cortier <bcortier@proton.me> Co-authored-by: Eliza Weisman <eliza@buoyant.io>

Cloud33 · 2023-05-15T01:39:15Z

This feature is great. Do you have a release schedule?

c98 · 2023-08-14T12:13:11Z

This feature is great. Do you have a release schedule?

I have the same concern.

# 0.2.3 (November 13, 2023) This release contains several new features. It also increases the minimum supported Rust version (MSRV) to Rust 1.63.0. ## Added - **rolling**: add option to automatically delete old log files (#2323) - **non_blocking**: allow worker thread name to be configured (#2365) - **rolling**: add a builder for constructing `RollingFileAppender`s (#2227) - **rolling**: add `Builder::filename_suffix` parameter (#2225) - **non_blocking**: remove `Sync` bound from writer for `NonBlocking` (#2607) - **non_blocking**: name spawned threads (#2219) ## Fixed - Fixed several documentation typos and issues (#2689, #2375) ## Changed - Increased minimum supported Rust version (MSRV) to 1.63.0+ (#2793) - Updated minimum `tracing-subscriber` version to [0.3.18][subscriber-v0.3.18] (#2790) [subscriber-v0.3.18]: https://github.com/tokio-rs/tracing/releases/tag/tracing-subscriber-0.3.18

# 0.2.3 (November 13, 2023) This release contains several new features. It also increases the minimum supported Rust version (MSRV) to Rust 1.63.0. ## Added - **rolling**: add option to automatically delete old log files (tokio-rs#2323) - **non_blocking**: allow worker thread name to be configured (tokio-rs#2365) - **rolling**: add a builder for constructing `RollingFileAppender`s (tokio-rs#2227) - **rolling**: add `Builder::filename_suffix` parameter (tokio-rs#2225) - **non_blocking**: remove `Sync` bound from writer for `NonBlocking` (tokio-rs#2607) - **non_blocking**: name spawned threads (tokio-rs#2219) ## Fixed - Fixed several documentation typos and issues (tokio-rs#2689, tokio-rs#2375) ## Changed - Increased minimum supported Rust version (MSRV) to 1.63.0+ (tokio-rs#2793) - Updated minimum `tracing-subscriber` version to [0.3.18][subscriber-v0.3.18] (tokio-rs#2790) [subscriber-v0.3.18]: https://github.com/tokio-rs/tracing/releases/tag/tracing-subscriber-0.3.18

max_log_files_feature

5a64008

ozgunozerk requested a review from a team as a code owner September 24, 2022 20:34

ozgunozerk mentioned this pull request Sep 24, 2022

new feature: keep only the last N log files in rotation #2284

Closed

hawkw approved these changes Sep 24, 2022

View reviewed changes

hawkw and others added 2 commits September 24, 2022 13:36

Merge branch 'master' into max-log-files

1dfa22a

doc fixes

67e1df1

hawkw reviewed Sep 24, 2022

View reviewed changes

tracing-appender/src/rolling/builder.rs Outdated Show resolved Hide resolved

hawkw requested changes Sep 24, 2022

View reviewed changes

add parsing feature for time crate

270efd3

hawkw approved these changes Sep 24, 2022

View reviewed changes

hawkw enabled auto-merge (squash) September 24, 2022 20:51

hawkw merged commit b7ef939 into tokio-rs:master Sep 24, 2022

hawkw mentioned this pull request Sep 24, 2022

backport recent changes from master #2324

Merged

ozgunozerk deleted the max-log-files branch September 25, 2022 07:43

flisky mentioned this pull request Feb 8, 2023

feat(appender): support brotli and gzip Compression #2461

Open

photino mentioned this pull request Feb 12, 2023

Keeping the most recent N log files zino-rs/zino#7

Closed

CBenoit mentioned this pull request Mar 4, 2023

Allow tracing-appender sized log and rotation #1940

Open

ozgunozerk mentioned this pull request May 8, 2023

Release notes is missing feature: tracing-subscriber 0.3.16 #2582

Closed

ffuugoo mentioned this pull request Aug 3, 2023

Replace env_logger with tracing qdrant/qdrant#2381

Merged

11 tasks

c98 mentioned this pull request Aug 14, 2023

tracing-appender: When the keep_last_n_files feature can be released? #2685

Open

davidbarsky mentioned this pull request Sep 26, 2023

chore: backport roughly a year's worth of changes #2728

Merged

dtoniolo mentioned this pull request Dec 7, 2023

Adds a check in tracing_appender #2826

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

max_log_files_feature #2323

max_log_files_feature #2323

ozgunozerk commented Sep 24, 2022

hawkw left a comment

hawkw left a comment

hawkw commented Sep 24, 2022

ozgunozerk commented Sep 24, 2022

hawkw left a comment

Cloud33 commented May 15, 2023

c98 commented Aug 14, 2023

max_log_files_feature #2323

max_log_files_feature #2323

Conversation

ozgunozerk commented Sep 24, 2022

Motivation

Solution

hawkw left a comment

Choose a reason for hiding this comment

hawkw left a comment

Choose a reason for hiding this comment

hawkw commented Sep 24, 2022

ozgunozerk commented Sep 24, 2022

hawkw left a comment

Choose a reason for hiding this comment

Cloud33 commented May 15, 2023

c98 commented Aug 14, 2023