Cancelling a crawl mid delayed-type-check orphans .delayed placeholders#496
Merged
Conversation
hts_wait_delayed() patches url_sav to the final name while a direct-to-disk transfer may still be writing to the .delayed placeholder path (bound when a back_wait tick lands between the slot's unlock and the patch, e.g. url_savename's post-stop test branch pumping sockets). From then on every cleanup keyed on IS_DELAYED_EXT(url_sav) is blind to the on-disk name and a cancelled mirror leaves X.N.delayed partials behind (#483). Move the placeholder to the final name right before the patch (back_delayed_rename, reopening a mid-transfer stream in append mode), and drop it in the cache-miss re-add path where a cancel between back_maydelete() and the re-add dropped its last reference. The .delayed leftover audit in local-crawl.sh now runs for every crawl (--skip-delayed-audit is gone), and 39_local-delayed-cancel exercises the cancel window: 4/10 runs failing on master, 10/10 clean with the fix; 0 orphans in 35 direct-harness cancelled crawls against ~1 in 6 before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Windows rename() fails when the target exists; unlinking newname there would destroy a previous run's completed file during an update crawl. Drop only what the slot owns: the placeholder if the rename failed, the moved partial if the reopen did. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Xavier Roche <roche@httrack.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cancelling a crawl mid delayed-type-check leaves
X.N.delayedpartials in the mirror (#483):hts_wait_delayed()patchesurl_savto the final name while the transfer can still be writing to the placeholder path, and every cleanup keyed onIS_DELAYED_EXT(url_sav)then misses the file. Fix:back_delayed_rename()moves the placeholder to the final name right before the patch; the cache-miss re-add path discards its placeholder when a cancel drops the last reference.The
.delayedleftover audit now runs for every local crawl, and the new39_local-delayed-canceltest pins the scenario (timing-dependent: 4/10 failing on master, 10/10 clean here; 0 orphans in 35 harness crawls vs about 1 in 6 before).Closes #483.