Remove the pre-3.31 (.dat/.ndx) cache import - #512
Merged
Conversation
The zip cache replaced this format in 3.31 (2003); what remained was an import-only parser for hostile input, compiled unconditionally, plus a tail of dead code keyed to it: the CACHE-1.5 writer behind if(1), the cross-session HTML-MD5 dedup (its store was never written in the zip era), the legacy legs of the startup rotation (which silently renamed or deleted a user's .dat/.ndx pair) and of hts_cache_reconcile. cache_init now detects a legacy pair, logs a clear refusal, leaves the files untouched, and re-crawls. The cache-legacy self-test pins refusal for both layouts (--update old.* and ro new.*), no is_update flag, and untouched files; proven end-to-end against the local server. The cacheindex self-test and fuzz-cachendx stay: cache_brstr/cache_binput still back the -#C cache listing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
xroche
force-pushed
the
remove-oldformat-cache
branch
from
July 9, 2026 19:45
71d39d5 to
4020cda
Compare
This was referenced Jul 26, 2026
xroche
added a commit
that referenced
this pull request
Jul 27, 2026
…does not (#764) The MD5 comparison the comment promises left the tree in two steps: #467 extracted the function out of htsparse.c's HT_ADD_END macro without the skip branch, and #512 removed the //[HTML-MD5]// cache entry it read. Drop the parenthetical; the write is unconditional. Signed-off-by: Xavier Roche <roche@httrack.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The zip cache replaced the
.dat/.ndxformat in 3.31 (2003). What remained was an import-only parser for hostile input, compiled unconditionally (the origin of #507's bounds fix and #510's original OOB), plus dead code keyed to it: the CACHE-1.5 writer behindif (1), the cross-session HTML-MD5 dedup whose store was never written in the zip era, and the legacy legs of the startup rotation andhts_cache_reconcile, which silently renamed or even deleted a user's.dat/.ndxpair. Net -1010 lines.On a pre-3.31 cache,
cache_initnow logs a clear refusal, leaves the files in place, and the site is re-crawled rather than aborting. The new-#test=cache-legacyself-test pins the refusal for both layouts (--updateold.* and read-only new.*) and that the files stay untouched; the reconcile self-test now pins that legacy pairs are left alone. Verified end-to-end against the local test server.cache_brstr/cache_binputand their cacheindex self-test/fuzzer stay: they still back the-#Ccache listing, which turns out to have been broken for zip caches since 3.31 (it only ever enumerated vianew.ndx; follow-up to file).