Resolve packs ~6 times faster than Git*

[Sebastian Thiel (Byron)]

*with 16 cores and SHA-256. This should scale, so I wouldn't be surprised to see even higher numbers with 64 cores.

And while what follows what LLM written, it's reviewed and tuned by me.

---

Operations that read an entire pack—such as cloning, fetching, indexing, connectivity checking, and pack verification—spend substantial time resolving delta-compressed objects.

gix now resolves these delta trees lazily and distributes branches across idle workers. This keeps cores productive even when a pack ends with one deep, expensive delta tree, without retaining an entire frontier of resolved bases.

Workload	Threads	Git	Git efficiency1	gix	gix efficiency	vs Git
Pathological phpstan2	1	304.51 s	100%	283.41 s	100%	1.07× faster
	8	65.41 s	58.2%	36.42 s	97.3%	1.80× faster
	16	52.79 s	36.1%	22.83 s	77.6%	2.31× faster
Bundled 7.6M-object Linux fixture3	1	156.65 s	100%	137.48 s	100%	1.14× faster
	16	42.97 s	22.8%	13.16 s	65.3%	3.27× faster
Private 10.9M-object Linux pack, SHA-14	8	125.11 s	—	30.60 s	86.8%	4.09× faster
Private 10.9M-object Linux pack, SHA-2565	1	87.85 s	100%	72.47 s	100%	1.21× faster
	16	48.68 s	11.3%	8.67 s	52.2%	5.61× faster

On the pathological pack, gix retains 97.3% efficiency at eight threads, compared with Git’s 58.2%. At 16 threads, gix retains 77.6% while Git falls to 36.1%.

The pattern holds across the other packs: gix keeps more useful work on each additional core, while Git’s wall-clock scaling flattens earlier.

Why SHA-256 is included

The SHA-1 and SHA-256 results come from the same private 10.9M-object Linux pack. The SHA-256 version preserves the compressed object data and delta topology byte-for-byte; only the pack trailer and index use SHA-256. This keeps scheduling and delta work constant, isolating the cost of hashing every reconstructed object.

Within gix, the SHA-256 resolver is 2.72×–2.99× faster than SHA-1 across the measured thread counts. This shows that hashing—not work distribution—is responsible for much of the remaining absolute SHA-1 cost. It also gives a preview of pack processing as repositories begin adopting Git’s SHA-256 object format.

For fairness, Git’s SHA-256 measurements use an OpenSSL-backed build. The portable backend in the original local Git build was substantially slower and would have exaggerated the difference.

All measurements were taken sequentially on the same ARM64 macOS host with 16 logical cores.

Footnotes

1. Efficiency is calculated from wall time as serial time / (thread count × parallel time). At 100%, doubling the cores halves wall time. Unlike resolver-only timing, this includes fixed command and index-processing overhead for both implementations. Git efficiency is unavailable for the private SHA-1 pack because its serial Git run was not recorded. ↩

2. The phpstan pack contains 100.7k objects and expands to 174.6 GB. Git’s 1/2/4/8/16-thread series used per-worker delta-base cache limits of 768/384/192/96/96 MiB, ensuring at least the default eight-thread aggregate allowance of 768 MiB. Commands: /usr/bin/time -lp git -c core.deltaBaseCacheLimit=<limit> index-pack --verify --threads=<threads> <pack> and /usr/bin/time -lp gix -t<threads> -v free pack verify <index>. ↩

3. This fixture contains 7.6M objects, expands to 95.6 GB, and is distributed with gitoxide as tests/fixtures/repos/linux.git/objects/pack/pack-3ee05b0f4e4c2cb59757c95c68e2d13c0a491289. Commands: /usr/bin/time -lp git index-pack --verify --threads=<threads> <pack> and /usr/bin/time -lp gix -t<threads> -v free pack verify <index>. ↩

4. This private 10.9M-object Linux pack is retained locally as a long-term benchmarking fixture and is not distributed with gitoxide. It expands to 146.7 GB and is identified locally as pack-6f64e91eccd8f68e174dc2c01416b392c66867e7. Git’s measurement used its default eight resolver threads. Commands: /usr/bin/time -lp git index-pack --verify --threads=8 <pack> and /usr/bin/time -lp gix --object-hash sha1 -t8 -v free pack verify <index>. ↩

5. The private SHA-256 fixture is intended for pack verification, not as a converted repository: commit and tree payloads still contain SHA-1 references. Git was built from cf5497b14c with SHA1_DC and OpenSSL 3.6.3’s SHA256_OPENSSL backend. Switching from its portable SHA256_BLK backend reduced Git’s eight-thread wall time from 144.15 to 40.45 seconds. Commands: /usr/bin/time -lp git index-pack --verify --object-format=sha256 --threads=<threads> <pack> and /usr/bin/time -lp gix --object-hash sha256 -t<threads> -v free pack verify <index>. ↩