Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for pruning based on file age #10

Closed
atc0005 opened this issue Sep 22, 2019 · 3 comments · Fixed by #27
Closed

Add support for pruning based on file age #10

atc0005 opened this issue Sep 22, 2019 · 3 comments · Fixed by #27
Labels
Milestone

Comments

@atc0005
Copy link
Owner

atc0005 commented Sep 22, 2019

Right now we're keeping X number of files and relying on filename patterns to help limit the files that are removed. To make this tool more useful, it will need to support age-based matching as well. Other criteria can be added later (e.g., size), but this one seems to be the more useful additoin to include first.

Match age in minutes, hours, days, ... ?

@atc0005 atc0005 added enhancement New feature or request command-line config labels Sep 22, 2019
@atc0005 atc0005 modified the milestones: Future, v0.3.0 Sep 22, 2019
@atc0005
Copy link
Owner Author

atc0005 commented Sep 26, 2019

Scratch notes.

Get current time, calculate based on provided flags, options. This will give us a time value. Then as we crawl Path, we will get modification time of files. If modTime is greater than calculated time, the file is a candidate.

start := time.Now()
... operation that takes 20 milliseconds ...
t := time.Now()
elapsed := t.Sub(start)

@atc0005
Copy link
Owner Author

atc0005 commented Sep 27, 2019

I think I have it, but may have additional polish to apply before I'm done with it.

@atc0005
Copy link
Owner Author

atc0005 commented Sep 27, 2019

The logic for keeping X many files needs to be reviewed. It made sense to have the prior behavior when we were not processing based on modification time (or even considering it at the code level), but now with a set of files of (intentionally) mixed age, the "keep X" logic is being applied to the matches, even when a large number of remaining files are in place.

Example:

ubuntu@testbox:/tmp$ ./elbow --path /tmp --pattern "reach-masterqa-" --keep 1 --recurse --keep-old --ignore-errors --age 120 --extension ".tmp" --extension ".war" --log-level info --log-format text --remove
INFO[0000] Starting evaluation of path                   extensions="[.tmp .war]" file_age=120 file_pattern=reach-masterqa- path=/tmp
INFO[0000] 12 files eligible for removal
INFO[0000] 1 files to keep as requested                  keep_oldest=true
INFO[0000] Ignoring file removal errors: true
INFO[0000] Removing filereach-masterqa-d9db6e2-20190501-1037.war  file=reach-masterqa-d9db6e2-20190501-1037.war removal_enabled=true
INFO[0000] Removing filereach-masterqa-1e14b45-20190507-1210.war  file=reach-masterqa-1e14b45-20190507-1210.war removal_enabled=true
INFO[0000] Removing filereach-masterqa-6e83c12-20190507-1239.war  file=reach-masterqa-6e83c12-20190507-1239.war removal_enabled=true
INFO[0000] Removing filereach-masterqa-a9ea1d0-20190508-1424.war  file=reach-masterqa-a9ea1d0-20190508-1424.war removal_enabled=true
INFO[0000] Removing filereach-masterqa-82a921c-20190510-1334.war  file=reach-masterqa-82a921c-20190510-1334.war removal_enabled=true
INFO[0000] Removing filereach-masterqa-a9ea1d0-20190510-1054.war  file=reach-masterqa-a9ea1d0-20190510-1054.war removal_enabled=true
INFO[0000] Removing filereach-masterqa-a9ea1d0-20190510-1321.war  file=reach-masterqa-a9ea1d0-20190510-1321.war removal_enabled=true
INFO[0000] Removing filereach-masterqa-a9ea1d0-20190510-1326.war  file=reach-masterqa-a9ea1d0-20190510-1326.war removal_enabled=true
INFO[0000] Removing filereach-masterqa-a9ea1d0-20190510-1316.war  file=reach-masterqa-a9ea1d0-20190510-1316.war removal_enabled=true
INFO[0000] Removing filereach-masterqa-c765a45-20190528-1255.war  file=reach-masterqa-c765a45-20190528-1255.war removal_enabled=true
INFO[0000] Removing filereach-masterqa-b1e3655-20190530-1021.war  file=reach-masterqa-b1e3655-20190530-1021.war removal_enabled=true
INFO[0000] 11 files successfully removed
INFO[0000] reach-masterqa-d9db6e2-20190501-1037.war      failed_removal=false
INFO[0000] reach-masterqa-1e14b45-20190507-1210.war      failed_removal=false
INFO[0000] reach-masterqa-6e83c12-20190507-1239.war      failed_removal=false
INFO[0000] reach-masterqa-a9ea1d0-20190508-1424.war      failed_removal=false
INFO[0000] reach-masterqa-82a921c-20190510-1334.war      failed_removal=false
INFO[0000] reach-masterqa-a9ea1d0-20190510-1054.war      failed_removal=false
INFO[0000] reach-masterqa-a9ea1d0-20190510-1321.war      failed_removal=false
INFO[0000] reach-masterqa-a9ea1d0-20190510-1326.war      failed_removal=false
INFO[0000] reach-masterqa-a9ea1d0-20190510-1316.war      failed_removal=false
INFO[0000] reach-masterqa-c765a45-20190528-1255.war      failed_removal=false
INFO[0000] reach-masterqa-b1e3655-20190530-1021.war      failed_removal=false
INFO[0000] 0 files failed to remove
INFO[0000] Elbow successfully completed.

These lines in particular emphasize my point:

INFO[0000] 12 files eligible for removal
INFO[0000] 1 files to keep as requested                  keep_oldest=true
INFO[0000] 11 files successfully removed

Since there were a larger number of files still remaining, shouldn't the "keep X" logic have been satisfied by their presence? Or, perhaps the current logic is best as it applies evenly across all matches?

atc0005 added a commit that referenced this issue Sep 28, 2019
- Refactor existing functions to use `hasMatching`
  prefix in an attempt to better spell out
  what makes matching values "valid"

- Add hasMatchingAge() function for comparing
  against user-provided age value (days)

- Update README to include new age-based threshold
  flag option

- Update CHANGELOG

refs #10
atc0005 added a commit that referenced this issue Sep 28, 2019
- Refactor existing functions to use `hasMatching`
  prefix in an attempt to better spell out
  what makes matching values "valid"

- Add hasMatchingAge() function for comparing
  against user-provided age value (days)

- Update README to include new age-based threshold
  flag option

- Update CHANGELOG

refs #10
atc0005 added a commit that referenced this issue Oct 15, 2019
Include additional fields added as part of implementing
support for pruning based on file age and other recent
issues.

refs #10, #38
atc0005 added a commit that referenced this issue Oct 15, 2019
Include additional fields added as part of implementing
support for pruning based on file age and other recent
issues.

Minor CHANGELOG update.

refs #10, #38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant