Keep the mirrored file when an --update re-fetch comes in short - #584
Merged
Conversation
An --update re-fetch that arrives incomplete (server declares a Content-Length it does not deliver, or the connection drops mid-body) is correctly refused: httrack logs "incomplete transfer ... will be retried on the next update" and does not store the partial body. But the incomplete-transfer branch in back_finalize() returned without noting the surviving copy, so the URL never reached new.lst and the end-of-update purge deleted the good file it had just declined to overwrite. filenote() the previous copy when it still exists on disk, mirroring the decode-failure path added in #560. Not compression-specific and needs no abort: a plain HTML page reproduces it, so it is broader than #521 (abort) and #557 (decode failure). Closes #562 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
The #562 /uptrunc/ routes copied upcodec's per-path body-fetch counter verbatim. Fold both onto one refetch_pass() helper and one dict. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An
--updatere-fetch that arrives incomplete (the server declares aContent-Lengthit never delivers, or the connection drops mid-body) is correctly refused: httrack logs "incomplete transfer ... will be retried on the next update" and keeps the partial out of the cache. It then deleted the good local copy anyway. The bytes on disk were never touched; the file died in the end-of-update purge, because the incomplete-transfer branch returned before noting the surviving copy, so the URL was absent fromnew.lstand the purge treated it like a page that had vanished from the site.The fix
filenote()s the previous copy when it still exists, the same guard #560 added on the decode-failure path. This one is broader: it needs no content coding and no abort, so a plain HTML page reproduces it (confirmed with-X0, which turns the purge off and lets the file survive). The new test mirrors a page and a direct-to-disk file, re-fetches both truncated under--update, and checks they keep their pass-1 content while a fully-served control still updates; reverting the fix deletes the files.Closes #562