Cache reconcile policy was triplicated and broken for zip caches - #491
Merged
Conversation
The old-vs-new hts-cache generation dance was written out three times: startup promote (htscoremain.c), interrupted-run keep-larger (htscoremain.c) and end-of-run rollback (htscore.c), with bare 32768/65536 thresholds. Fold them into one policy function in htscache.c, selected by mode, with named thresholds. Behavior-preserving, with one provably-dead branch dropped: the interrupted-run legacy .dat arm sat in the else of fexist(new.dat) while itself requiring new.dat, so it could never run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The extracted policies carried format gates mangled when the zip cache landed: - promote: the legacy .dat/.ndx arm hid in the else of 'new.zip exists', so a pure-legacy cache never got its old generation promoted; - interrupted-run: the zip arm was gated on fexist(new.dat), never true for a modern cache, so the whole site was a no-op; - rollback: only .dat/.ndx were restored. On a zip cache the restore was a no-op, so a transient outage left the thin error cache as new.zip and the next run's rotation deleted old.zip: the good generation was lost and everything got re-downloaded. Handle the two formats independently in each mode, and restore the .lst/.txt sidecars regardless of format. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
-#test=reconcile drives hts_cache_reconcile() over sized file fixtures for the three modes; the cases covering the fixed gates fail against the old code (13 checks). 37_local-cache-outage crawls, stops the server, re-runs the mirror and asserts the previous new.zip comes back byte-identical with no old.zip left behind (local-crawl.sh grows --rerun-dead for this); it fails against the old rollback too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The audit flagged the rollback trigger (stat_files<=0 && recv<32K) as misfiring on a legit all-304 small-site update. Probing shows it does not: stat_files counts cache-carried files too (even -p0 reports them written), so the trigger is unreachable for any run that scanned links, and only real failures reach it. No engine change; pin the contract instead: 38_local-update-304 updates a tiny fully-cacheable site (/mini304/, served through the /big/ 304 validator) and asserts no rollback notice, a 'no files updated' summary, and intact files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
fsize() returns -1 for a missing file, which passes the 'new < TINY' size test, so an interrupted-run reconcile could promote a solid old generation onto an absent new one. That is unreachable today (PROMOTE runs first and normalizes the missing-new case), but it is a latent sharp edge a future reordering would expose. Require the new file to exist before the size comparison, on both the zip and legacy arms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The old/new hts-cache generation policy was copy-pasted across three call sites with divergent magic thresholds, and its zip-era format gates were broken. In the worst case a failed update deleted the only good cache and forced a full re-download. This folds the three copies into one hts_cache_reconcile() and fixes the gates, covered by a new -#test=reconcile self-test and a dead-server update test.
Probing two sibling audit findings (the delete_old purge match and the all-304 rollback) showed neither reproduces; the second is pinned by a test.