http: import upstream packfile-URI concurrency fixes - #28
Open
ttaylorr-oai wants to merge 6 commits into
Open
Conversation
abg-OAI
approved these changes
Aug 8, 2026
ttaylorr-oai
marked this pull request as ready for review
August 8, 2026 17:43
The --packfile mode accepts one --index-pack-arg=<arg> option per argument passed to index-pack, but its documentation and option dependency errors still refer to the plural --index-pack-args form. Correct the spelling and describe the repeatable per-argument form. Signed-off-by: Ted Nyman <tnyman@openai.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
finish_http_pack_request() passes its staging-file descriptor to index-pack through child_process.in. start_command() takes ownership of a supplied descriptor and closes it, even when starting the child fails. Do not close the descriptor again after run_command() returns. Signed-off-by: Ted Nyman <tnyman@openai.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
A resumed pack request may already have all bytes of the remote pack. A server can respond to the resulting Range request with HTTP 416 instead of returning an empty response. Accept that response in each pack-download caller and let index-pack validate the completed staging file. This can happen without concurrent downloads when a previous attempt completed the transfer but failed before indexing it. Add a regression test that seeds a complete partial pack and checks that http-fetch indexes it after the server returns HTTP 416. Signed-off-by: Ted Nyman <tnyman@openai.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Pack requests stage downloads in a predictable partial-pack file so an interrupted transfer can be resumed. Both packfile URI and ordinary dumb HTTP requests use this staging path. Opening it in append mode forces each write to the current end of the file, so concurrent responses can append duplicate data and corrupt the pack. Open the partial pack read-write without O_APPEND and seek once to its current end. Each downloader then retains the offset matching the Range it requested. Because the staging key must uniquely identify immutable pack contents, overlapping responses write the same bytes at the same offsets instead of extending the file with duplicate data. Duplicate the staging descriptor for index-pack instead of reopening the path after closing the stream. Another downloader may unlink the staging path before indexing begins, but index-pack can still read the retained descriptor. Exercise resumed transfers and overlapping 200 and 206 responses, and clarify the staging-key documentation. Signed-off-by: Ted Nyman <tnyman@openai.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
On Windows, an open file must permit FILE_SHARE_DELETE before another process can unlink it. MinGW's non-append O_RDWR open enables that sharing mode only for an existing file; adding O_CREAT falls back to _wopen(), which cannot set it. First try opening the partial pack without O_CREAT. If it does not exist, create it exclusively, close that descriptor, and retry through the existing-file path. A racing creator retries after EEXIST. This ensures that every retained descriptor permits another downloader to unlink the staging path. Add an unlink-while-indexing test that does not require FIFOs and can therefore run on MinGW. Signed-off-by: Ted Nyman <tnyman@openai.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
When index-pack finds an existing keep file it reports pack rather than keep. Accept either result from http-fetch, and only register a keep lockfile when this fetch created it. Read the pack/keep prefix and hash without consuming any following fsck output, validate the reported pack hash against the advertised hash, and exercise a packfile URI fetch with a pre-existing keep file. Signed-off-by: Ted Nyman <tnyman@openai.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
ttaylorr-oai
force-pushed
the
tb/codex/packfile-uri-concurrency
branch
from
August 8, 2026 17:57
c4244bd to
4cc8b32
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This carries Ted Nyman's six-commit
tn/packfile-uri-concurrencytopic, as merged into Git'sjchat 8004c4e. The topic is rebased onto the currentopenai/gitmasterso Codex v3 can infermasteras its lane boundary. Downstream head is4cc8b3223b; the upstream source tip is c4244bd.git range-diffreports all six patches unchanged.The series fixes resumed and concurrent packfile-URI downloads: it accepts a complete partial pack after HTTP 416, avoids concurrent append corruption, keeps indexing safe across unlink/Windows sharing behavior, and preserves a pre-existing
.keepfile. It also fixes the--index-pack-argdocumentation and a double-close in the HTTP pack path.Upstream commits:
http-fetch: correct --index-pack-arg documentationhttp: avoid closing index-pack input twicehttp: accept HTTP 416 for complete partial packshttp: avoid concurrent appends to partial packshttp: permit unlinking partial packs on Windowsfetch-pack: accept "pack" output for packfile URIsValidation:
make -j8 DEVELOPER=1 git git-http-fetch git-http-pushGIT_TEST_HTTPD=true make -C t t5550-http-fetch-dumb.sh(63/63)GIT_TEST_HTTPD=true make -C t t5702-protocol-v2.sh(86/86)git range-diff 55526a1..c4244bd origin/master..HEAD(six patch-equivalent commits)git diff --check origin/master..HEADgit merge-tree --write-tree origin/codex HEADThis PR is the reviewed topic artifact. The Codex v3 plan/rebuild flow, rather than a direct merge of this PR, enrolls it into
codex.