Add live reload of exclusion lists (and switch to go 1.25)#428
Add live reload of exclusion lists (and switch to go 1.25)#428yzqzss merged 13 commits intointernetarchive:mainfrom
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #428 +/- ##
==========================================
- Coverage 55.58% 54.77% -0.82%
==========================================
Files 119 120 +1
Lines 7313 7343 +30
==========================================
- Hits 4065 4022 -43
- Misses 2924 2994 +70
- Partials 324 327 +3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull Request Overview
This PR implements live reloading functionality for exclusion files, allowing the application to automatically reload exclusion patterns without requiring a restart. The feature is disabled by default to avoid breaking changes and can be enabled via command-line flags.
- Adds live reload capability for exclusion files with configurable intervals (default 60 seconds)
- Refactors exclusion regex handling to use atomic operations for thread-safe updates
- Introduces graceful shutdown handling for the live reload goroutine
Reviewed Changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/pkg/config/exclusions.go | New file containing exclusion file loading and regex compilation logic |
| internal/pkg/config/config.go | Refactored to use atomic storage for exclusion regexes and added live reload goroutine |
| internal/pkg/preprocessor/preprocessor.go | Updated to use new thread-safe getter method for exclusion regexes |
| internal/pkg/archiver/archiver.go | Added config context cancellation during shutdown |
| internal/pkg/log/log.go | Minor whitespace cleanup |
| cmd/get.go | Added command-line flags for live reload configuration |
This PR adds the capability to "live reload" the exclusion files.
It is off by default, I would like the opposite but I didn't want to implement a breaking change, we can discuss it if someone thinks it should be on by default.
There is a new setting
--exclusion-file-live-reloadto enable it, by default it reloads the files every 60 seconds, but this can be changed with--exclusion-file-live-reload-intervalwhich takes a int corresponding to the number of seconds between each reload.It works with both file on disk, and remotely. It's a very simple implementation, there are ways we could make it more efficient, it will certainly come in a future PR when I have more time to allow to it.