One bad cache entry ends the whole mirror - #494
Merged
Conversation
Users read "file not stored in cache due to bogus state" as a crash or cache corruption; it is a benign skip when the transfer is shorter than the Content-Length. Say what happened, the consequence, and the -%B override instead, and give the delayed-type variant a plain wording. Test pins updated; 22_local-broken-size gains set -e (its first crawl's audit failure was masked by the second crawl's exit status). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Since #426 any new.zip write failure cleanly aborts the whole mirror. Keep that for storage-level trouble (fatal errno such as ENOSPC, or every entry failing), but let an isolated failure drop only the current entry: abandon it, warn with the URL, and keep the mirror and the cache stream going. A streak of CACHE_MAX_WRITE_FAILURES consecutive failures still aborts. Also degrade the >2GB assertf crash in cache_add: an oversized on-disk body is stored headers-only (X-In-Cache: 0), an in-memory one drops the entry. The cache-writefail self-test now pins all four regimes (fatal errno, persistent streak, isolated skip with sibling round-trip, oversize); on the previous code it fails the new assertions and hits the oversize assertf (SIGABRT). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Adversarial test audit found the policy's core semantics unpinned: no test distinguished a consecutive-failure cap from a total count, or proved a stored entry resets the streak. Phase 1 now asserts the abort lands exactly on the 8th consecutive failure, and a new phase drives 10 failures interleaved with successes and asserts no abort. The .test now greps a URL-bearing skip warning. check_fatal_io_errno gains EDQUOT (quota exhaustion is disk-full for our purposes). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since #426 a failing new.zip write aborts the mirror cleanly instead of crashing, but any failure aborts: one oversized or transiently unwritable entry ends the whole crawl. Storage-level trouble (a fatal errno such as ENOSPC, or a run of consecutive failures) still aborts; anything else now drops just the failing entry with a warning naming the URL, and the file stays on disk so the next update re-fetches it. The >2GB assertf in cache_add (reachable with -k) degrades the same way: on-disk bodies are stored headers-only, in-memory ones are skipped. The cache-writefail self-test covers all four regimes and fails on the previous code (SIGABRT on the oversize assertf).
Also rewords the "bogus state" warnings that users have read as a crash for a decade: the size-mismatch skip now says what happened and points at -%B.