Repository navigation
pr-2239/spkrka/krka/incremental-commit-graph-v3
tagged this
07 Oct 14:22
When fetch.writeCommitGraph is enabled, the commit-graph is rebuilt from all reachable refs after every fetch. This is unnecessarily expensive on repositories with many refs, since add_ref_to_set() validates each ref against the odb. This series optimizes the commit-graph write by using only the newly updated refs as seeds instead of scanning all refs. Since fetch writes the commit-graph in split mode, the newly fetched history is added as a new layer on top of the existing chain. A three-mode enum (REACHABLE / TIPS / SKIP) makes the policy explicit: * No-op fetch: skip the commit-graph write entirely * Updated refs + existing graph: write incrementally from updated tips only * No existing graph or multi-remote fetch: fall back to full reachable scan Patch 1 adds a commit-info subcommand to test-tool read-graph for verifying graph contents in tests. Patch 2 implements the optimization in builtin/fetch.c with tests covering the incremental, unrelated-commit, no-op, fallback, and shallow-rejected cases. Benchmark on a synthetic setup: git.git with 200K extra packed refs (~206K total), a local file:// remote, an existing split commit-graph and a warm page cache. Times are the median of 9 runs of the trace2 region fetch/write-commit-graph: scenario before after no-op fetch 380 ms (skipped) 1 ref updated 357 ms 9.3 ms 10 refs updated 359 ms 8.9 ms Changes since v2: * Trim the commit message: inline the commit references, keep the explanation of why the incremental write relies on split mode, and drop the paragraphs that only restated the diff (collecting the tips, auto-followed tags, skipping shallow-rejected refs). * Reword the comment on skipping shallow-rejected refs without the reference to store_updated_refs(). * Simplify the t5537 test by using "test_commit -C ... --no-tag" instead of subshells. Changes since v1: * Explain in the commit message that fetch writes in split mode, so the new tips are added as a new layer on top of the existing chain, and that the incremental path relies on this (a non-split write would replace the graph with only the closure of the seeds). * Extend the incremental test to check that a local-only commit, which was in the graph before the fetch but is not reachable from the fetched tips, is still in the graph afterwards. * Add benchmark numbers to the commit message. * Add a comment explaining why shallow-rejected refs are skipped when collecting the updated tips (like store_updated_refs(), since their history is incomplete), and mention it in the commit message. * Add a test in t5537 for a fetch with fetch.writeCommitGraph where a ref is rejected because it would require changes to .git/shallow. Without the check, the commit-graph write dies on the missing parent. Kristofer Karlsson (2): test-tool read-graph: add commit-info subcommand fetch: write commit-graph using updated refs only builtin/fetch.c | 69 +++++++++++++++++++++++++++++++++----- commit-graph.c | 2 +- commit-graph.h | 1 + t/helper/test-read-graph.c | 23 ++++++++++++- t/t5510-fetch.sh | 59 ++++++++++++++++++++++++++++++++ t/t5537-fetch-shallow.sh | 22 ++++++++++++ 6 files changed, 165 insertions(+), 11 deletions(-) base-commit: 0f8e75abebff0877cae681a3d5ff31ac47f54220 Submitted-As: https://lore.kernel.org/git/pull.2239.v3.git.1791382977.gitgitgadget@gmail.com In-Reply-To: https://lore.kernel.org/git/pull.2239.git.1790930019.gitgitgadget@gmail.com In-Reply-To: https://lore.kernel.org/git/pull.2239.v2.git.1791279992.gitgitgadget@gmail.com