Skip to content

Add live reload of exclusion lists (and switch to go 1.25)#428

Merged
yzqzss merged 13 commits intointernetarchive:mainfrom
CorentinB:exclusions-live-reload
Aug 14, 2025
Merged

Add live reload of exclusion lists (and switch to go 1.25)#428
yzqzss merged 13 commits intointernetarchive:mainfrom
CorentinB:exclusions-live-reload

Conversation

@CorentinB
Copy link
Copy Markdown
Collaborator

This PR adds the capability to "live reload" the exclusion files.

It is off by default, I would like the opposite but I didn't want to implement a breaking change, we can discuss it if someone thinks it should be on by default.

There is a new setting --exclusion-file-live-reload to enable it, by default it reloads the files every 60 seconds, but this can be changed with --exclusion-file-live-reload-interval which takes a int corresponding to the number of seconds between each reload.

It works with both file on disk, and remotely. It's a very simple implementation, there are ways we could make it more efficient, it will certainly come in a future PR when I have more time to allow to it.

@CorentinB CorentinB self-assigned this Aug 12, 2025
@CorentinB CorentinB added the enhancement New feature or request label Aug 12, 2025

This comment was marked as outdated.

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Aug 12, 2025

Codecov Report

❌ Patch coverage is 17.12329% with 121 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.77%. Comparing base (011c133) to head (b865524).
⚠️ Report is 113 commits behind head on main.

Files with missing lines Patch % Lines
internal/pkg/config/exclusions.go 1.98% 99 Missing ⚠️
internal/pkg/config/config.go 41.66% 18 Missing and 3 partials ⚠️
internal/pkg/preprocessor/preprocessor.go 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #428      +/-   ##
==========================================
- Coverage   55.58%   54.77%   -0.82%     
==========================================
  Files         119      120       +1     
  Lines        7313     7343      +30     
==========================================
- Hits         4065     4022      -43     
- Misses       2924     2994      +70     
- Partials      324      327       +3     
Flag Coverage Δ
e2etests 37.29% <15.75%> (-0.20%) ⬇️
unittests 30.85% <6.16%> (-1.09%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@CorentinB CorentinB requested a review from Copilot August 12, 2025 15:54

This comment was marked as outdated.

@CorentinB CorentinB requested a review from Copilot August 12, 2025 16:07

This comment was marked as outdated.

CorentinB and others added 4 commits August 12, 2025 18:12
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@CorentinB CorentinB requested a review from Copilot August 12, 2025 16:19
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements live reloading functionality for exclusion files, allowing the application to automatically reload exclusion patterns without requiring a restart. The feature is disabled by default to avoid breaking changes and can be enabled via command-line flags.

  • Adds live reload capability for exclusion files with configurable intervals (default 60 seconds)
  • Refactors exclusion regex handling to use atomic operations for thread-safe updates
  • Introduces graceful shutdown handling for the live reload goroutine

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
internal/pkg/config/exclusions.go New file containing exclusion file loading and regex compilation logic
internal/pkg/config/config.go Refactored to use atomic storage for exclusion regexes and added live reload goroutine
internal/pkg/preprocessor/preprocessor.go Updated to use new thread-safe getter method for exclusion regexes
internal/pkg/archiver/archiver.go Added config context cancellation during shutdown
internal/pkg/log/log.go Minor whitespace cleanup
cmd/get.go Added command-line flags for live reload configuration

Comment thread internal/pkg/config/config.go
Comment thread internal/pkg/config/config.go Outdated
Comment thread internal/pkg/config/config.go Outdated
Comment thread internal/pkg/config/config.go Outdated
Comment thread cmd/get.go Outdated
@CorentinB CorentinB requested a review from willmhowes August 13, 2025 16:50
Comment thread internal/pkg/config/config.go Outdated
@CorentinB CorentinB requested a review from yzqzss August 14, 2025 07:51
@CorentinB CorentinB changed the title Add live reload of exclusion lists Add live reload of exclusion lists (and switch to go 1.25) Aug 14, 2025
Comment thread internal/pkg/config/exclusions.go Outdated
Comment thread internal/pkg/config/config.go Outdated
@CorentinB CorentinB requested a review from yzqzss August 14, 2025 08:34
@yzqzss yzqzss merged commit c105a01 into internetarchive:main Aug 14, 2025
2 checks passed
@CorentinB CorentinB deleted the exclusions-live-reload branch August 14, 2025 09:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants