Serve the cache from local disk and sync with the NFS archive - #5
Merged
Conversation
Serving nginx's proxy_cache from the NFS volume costs an open/stat round trip per hit and a metadata walk over millions of entries for the cache manager, and nginx cannot share one cache directory between instances anyway. /var/cache/nginx is now local node disk; /cache stays the shared volume but as an archive that every instance restores from in the background at start (newest entries first, nginx already serving) and backs up into on a daily crond schedule with per-container jitter and an mkdir lock on the archive. Both directions use rsync --update and never delete, so the archive is the union of every instance's cache with the newest copy of each entry winning. Cache entries are MD5-named immutable blobs, which is what makes that merge rule sufficient. In-flight temp files (<hash>.<counter>) are filtered out so a partial body can never be copied. The scripts drop to the nginx user because the archive is root-squashed NFS. nginx serves a cache file that appears on disk after startup (a lookup that misses the in-memory index still opens and validates the file), so startup does not wait for a ~1 TB copy; /status reports restore and backup progress under "archive".
The startup test now mounts a one-entry fake archive at /cache and checks that the entry lands in the local cache, that the backup is scheduled, that /status carries the archive block and that a manual backup runs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
/var/cache/nginxis now local node disk;/cachestays the shared NFS volume but as an archive rather than the live cache. Every instance restores from it in the background at start (newest entries first, nginx already serving) and backs up into it on a dailycrondschedule (default 03:00 plus a deterministic per-container jitter of up to 120 min,mkdirlock on the archive so overlapping runs serialise). Both directions usersync --updateand never delete, so the archive is the union of every instance's cache with the newest copy of each entry winning. In-flight temp files (<hash>.<counter>) are filtered so a partial body can never be copied./statusgains anarchiveblock with restore/backup progress;purge-cached-404s.sh --archivepurges the archive too. New:cache-lib.sh,cache-restore.sh,cache-backup.sh,test/cache-sync-test.sh(run at image build).Why
Serving
proxy_cachefrom NFS costs an open/stat round trip per hit and a metadata walk over millions of entries for the cache manager, and nginx cannot share one cache directory between instances anyway (the index is per-process shared memory). This lets us run several instances and takes the read traffic off the NAS. A blocking restore was ruled out: ~1 TB / millions of files takes hours, and nginx serves cache files that appear on disk after startup (verified on 1.26), so warming in the background costs nothing but cold misses.How to test
sh test/cache-sync-test.shlocally; the image build runs it under BusyBox ash. CI mounts a one-entry fake archive at/cache, checks the entry lands in the local cache, that the backup is scheduled, that/statuscarries thearchiveblock and that a manualcache-backup.sh --fullruns. To deploy: change the stack's volumes to/data/owl-cache:/var/cache/nginxand/cache:/cache, setCACHE_MAX_SIZEto the node disk, and watchcurl /status | jq .archive.restoreuntilstateisdone.Follow-ups
Nodes and NAS must agree on time (NTP) for newest-wins to be meaningful. The
X-Force-Refreshwarm-up only warms the instance it hits: warm one, runcache-backup.sh, let the others pick it up on restore. If the 11 TB local disk turns out to persist across container restarts, the.restoredmarker already makes subsequent starts skip the restore. The CI-workflow commit is applied separately from the local checkout (PAT lacksworkflowscope).