When --update re-fetches an already-mirrored URL and the transfer comes up short (the server declares a Content-Length it does not deliver, or the connection drops mid-body), httrack correctly refuses to store the partial body and logs:
incomplete transfer (expected 42 bytes, got 21): file not cached, will be retried on the next update
and then deletes the good local copy anyway.
The mirrored bytes themselves are never touched. The file dies in the end-of-session purge (opt->delete_old, on by default, htscore.c:2081): the purge removes every path listed in hts-cache/old.lst but absent from hts-cache/new.lst, and the incomplete-transfer branch in back_finalize() (htsback.c:614-631) returns early, before anything calls filenote() for that URL. Nothing ever writes it into new.lst, so the purge treats a file that failed to transfer exactly like a file that vanished from the site.
Reproduced on master (9ec5a3a) with a local server that serves a page in full on the first fetch and then, on the second, declares the same Content-Length while writing half the body:
- Fresh crawl mirrors
trunc.html.
--update re-crawl: the re-fetch is incomplete, httrack logs the message above, and the mirror keeps the good pass-1 bytes on disk.- End of session:
old.lst lists trunc.html, new.lst does not, and the purge deletes it. Running the same update with -X0 (purge off) keeps the file with its pass-1 content, which pins the mechanism.
This is not specific to compressed bodies (the plain-HTML repro above has no Content-Encoding), and it needs no abort, so it is broader than #521 and #557. It is the same data-loss family: a failed re-fetch must not destroy the copy it failed to refresh.
Fix shape: note the surviving copy before the early return, the way the decode-failure path in #560 now does.
When
--updatere-fetches an already-mirrored URL and the transfer comes up short (the server declares aContent-Lengthit does not deliver, or the connection drops mid-body), httrack correctly refuses to store the partial body and logs:and then deletes the good local copy anyway.
The mirrored bytes themselves are never touched. The file dies in the end-of-session purge (
opt->delete_old, on by default,htscore.c:2081): the purge removes every path listed inhts-cache/old.lstbut absent fromhts-cache/new.lst, and the incomplete-transfer branch inback_finalize()(htsback.c:614-631) returns early, before anything callsfilenote()for that URL. Nothing ever writes it intonew.lst, so the purge treats a file that failed to transfer exactly like a file that vanished from the site.Reproduced on master (9ec5a3a) with a local server that serves a page in full on the first fetch and then, on the second, declares the same
Content-Lengthwhile writing half the body:trunc.html.--updatere-crawl: the re-fetch is incomplete, httrack logs the message above, and the mirror keeps the good pass-1 bytes on disk.old.lstliststrunc.html,new.lstdoes not, and the purge deletes it. Running the same update with-X0(purge off) keeps the file with its pass-1 content, which pins the mechanism.This is not specific to compressed bodies (the plain-HTML repro above has no
Content-Encoding), and it needs no abort, so it is broader than #521 and #557. It is the same data-loss family: a failed re-fetch must not destroy the copy it failed to refresh.Fix shape: note the surviving copy before the early return, the way the decode-failure path in #560 now does.