🎉Introducing gix archive 🎉

[Byron]

gix archive is a new sub-command which pretty much does what you think it will: given a treeish and a file path, it will extract the treeish exactly like it would when checking it out to stream it into the file path in one of multiple formats, tar, tar.gz and zip.

gix archive some advantages over git archive:

• it supports streaming large files like those coming in with git lfs (zip only)
• it's meaningfully faster
• it consumes less memory (consistently only with tar)
• (near future: submodule support)

Performance

Let's dig into some performance comparisons on the linux kernel cloned from git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git.

time	git	gix	factor
tar	4.52s	3.18s	1.42x
tar.gz	26.20s	11.56s	2.26x
tar.gz min	11.08s	4.18s	2.65x
tar.gz max	45.21s	21.32s	2.12x
zip	25.90s	11.38s	2.19x

The reason for generally better performance is certainly that there is very basic concurrency that offloads the streaming of worktree data into its own thread. That way, the container format can be handled by the main thread all by itself. Further, gix is has a faster odb implementation and uses zlib-ng which overall yields about 10% better performance when reading objects from the object store.
When compressing, gix also benefits from the flate2 crate and its advanced backends.

It's interesting to see that thanks to gix it's now possible to get the highest compression level (tar.gz max) in the same time that git takes for the default level. In this particular case, one only saves ~3MB with that though. Also, one can get a lightly compressed archive (tar.gz min) faster than an uncompressed tar archive from git, albeit at low compression with a 345MB size.

File Sizes

All the performance in the world wouldn't be useful if the produced files would be considerably bigger. Let's take a look.

filesize	git	gix	% smaller
tar	1.375GB	1372GB	0.2%
tar.gz	223.1MB	222.9MB	0.1%
zip	267.5MB	265.4MB	0.8%

When looking at tar.gz, it seems to be quite exactly the same as what git produces and is quite uninteresting in that.

zip, however, takes the lead in being slightly smaller and more than twice as fast to produce. Unfortunately it still has a shortcoming of not being able to reproduce symbolic links. If that would be fixed, it would be the most advanced format as it's also able to stream large files.

Memory Consumption

Finally, let's be sure that gix doesn't need unreasonable amounts of memory producing these files.

max-res	git	gix	% smaller
tar	1.245GB	1.063GB	15%
tar.gz	1.493GB	1.237GB	20%
zip	1.621GB	1.276GB	21%

peak	git	gix	% bigger
tar	160MB	154MB	-4%
tar.gz	172	258	50%
zip	167	247	48%

When looking at max-resident (max-res) size, gix uses consistently less, saving nearly 20% at all times. However, when looking at the peak memory (which probably doesn't include virtual memory), gix uses nearly 50% more. This clearly has to do with the container formats which seem to keep quite a lot of extra data around when setting them up, which might be an issue in repositories with a lot of files under the assumption that this scales with file-count.

It's worth noting that despite some shortcomings, it seems that tar is the best format when memory consumption is a concern - then gix will always outperform git both in memory consumption and performance (*in this particular setup).

Shortcomings

However, it's not all roses right now, and probably due to me creating the zip archive incorrectly, symlinks for some reason don't manifest during extraction despite being contained in the archive.
This works when using tar though.

Further, I have the feeling that compression settings aren't applied for some reason for tar.gz, and it's unclear how to set the compression for zip archives.

Also, submodules aren't yet added to the archive, which is the same shortcoming as for git itself, but that bound to happen as submodule support is currently being added to gitoxide.

Conclusion

Implementing a minimal viable product ofgix archive merely as an experiment took only 3 days and showed how powerful gix has become, making it possible to write tools that don't only rival the standard implementation, but can even surpass many aspects of it.
I will work hard to reach feature parity with git2 for starters and then do my best to make it easier for users to choose gix over git2 when starting new projects.

Q & A

Q: Can I use gix archive instead of git archive?

In think it's worth giving it a short if you need the extra performance or the extra capabilities. Be aware of the current shortcomings though, and maybe even contribute a fix.

Q: Why does gix archive exist?

As gix is a development tool to be able to run the gix crate in the real world, it made sense to be able to test the worktree related code in a context that doesn't involve writing files to disk. Implementing this means we need to be flexible enough to be able to put all related parts together in different ways, and gix archive is a very nice application of said 'worktree machinery'.

As it turns out, many folks wanted it to support submodules as well, and with this work ongoing it seems similarly useful to validate the gix API against such a need - supporting submodules should be reasonably easy with anything gix comes up with.

Q: Can I use this in my own crate?

Yes, there is gix-archive that implements archiving, and gix-worktree-stream that provides a stream of entries that would make up the worktree on disk, bit for bit.

Q: Could the same be implemented with git2?

Definitely, and I'd be keen to see such an implementation in comparison to git and gix!

Data

Versions used initially.

✦ ❯ git --version && gix --version
git version 2.39.2 (Apple Git-143)
gix v0.28.0-77-g231ac1c6a

Then after updates to how gz compression works, it's this one:

❯ gix --version
gix v0.28.0-102-gc7d912917

archive -f tar

✦ ❯ gix rev parse @
a1effab7a3a35a837dd9d2b974a1bc4939df1ad5

✦ ❯ hyperfine --warmup 1 'git archive --format tar @ >/dev/null' 'gix archive -f tar /dev/null'
Benchmark 1: git archive --format tar @ >/dev/null
  Time (mean ± σ):      4.520 s ±  0.073 s    [User: 4.190 s, System: 0.318 s]
  Range (min … max):    4.449 s …  4.676 s    10 runs

Benchmark 2: gix archive -f tar /dev/null
  Time (mean ± σ):      3.184 s ±  0.014 s    [User: 3.291 s, System: 0.221 s]
  Range (min … max):    3.168 s …  3.216 s    10 runs

Summary
  gix archive -f tar /dev/null ran
    1.42 ± 0.02 times faster than git archive --format tar @ >/dev/null

archive -f tar-gz

with libflate

✦ ❯ hyperfine --warmup 1 'git archive --format tar.gz @ >/dev/null' 'gix archive -f tar-gz /dev/null'
Benchmark 1: git archive --format tar.gz @ >/dev/null
  Time (mean ± σ):     26.200 s ±  0.138 s    [User: 25.675 s, System: 0.505 s]
  Range (min … max):   26.037 s … 26.469 s    10 runs

Benchmark 2: gix archive -f tar-gz /dev/null
  Time (mean ± σ):     21.595 s ±  0.037 s    [User: 24.307 s, System: 0.283 s]
  Range (min … max):   21.551 s … 21.673 s    10 runs

Summary
  gix archive -f tar-gz /dev/null ran
    1.21 ± 0.01 times faster than git archive --format tar.gz @ >/dev/null

with flate2

✦ ❯ hyperfine --warmup 1  'gix archive -f tar-gz /dev/null'
Benchmark 1: gix archive -f tar-gz /dev/null
  Time (mean ± σ):     11.565 s ±  0.152 s    [User: 14.514 s, System: 0.323 s]
  Range (min … max):   11.455 s … 11.982 s    10 runs

with flate2 level 9

✦ ❯ hyperfine --warmup 1 'git archive -9 --format tar.gz @ >/dev/null' 'gix archive -f tar-gz -l9 /dev/null'
Benchmark 1: git archive -9 --format tar.gz @ >/dev/null
  Time (mean ± σ):     45.318 s ±  0.117 s    [User: 44.596 s, System: 0.687 s]
  Range (min … max):   45.178 s … 45.486 s    10 runs

Benchmark 2: gix archive -f tar-gz -l9 /dev/null
  Time (mean ± σ):     21.320 s ±  0.255 s    [User: 24.296 s, System: 0.362 s]
  Range (min … max):   21.183 s … 22.014 s    10 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary
  gix archive -f tar-gz -l9 /dev/null ran
    2.13 ± 0.03 times faster than git archive -9 --format tar.gz @ >/dev/null

with flate2 level 1

✦ ❯ hyperfine --warmup 1 'git archive -1 --format tar.gz @ >/dev/null' 'gix archive -f tar-gz -l1 /dev/null'
Benchmark 1: git archive -1 --format tar.gz @ >/dev/null
  Time (mean ± σ):     11.078 s ±  0.131 s    [User: 10.642 s, System: 0.388 s]
  Range (min … max):   10.904 s … 11.397 s    10 runs

Benchmark 2: gix archive -f tar-gz -l1 /dev/null
  Time (mean ± σ):      4.181 s ±  0.014 s    [User: 7.008 s, System: 0.173 s]
  Range (min … max):    4.164 s …  4.210 s    10 runs

Summary
  gix archive -f tar-gz -l1 /dev/null ran
    2.65 ± 0.03 times faster than git archive -1 --format tar.gz @ >/dev/null

archive -f zip

✦ ❯ hyperfine --warmup 1 'git archive --format zip @ >/dev/null' 'gix archive -f zip /dev/null' && hyperfine --warmup 1 'git archive --format tar.gz @ >/dev/null' 'gix archive -f tar.gz /dev/null'
Benchmark 1: git archive --format zip @ >/dev/null
  Time (mean ± σ):     24.901 s ±  0.998 s    [User: 22.878 s, System: 2.001 s]
  Range (min … max):   23.486 s … 26.243 s    10 runs

Benchmark 2: gix archive -f zip /dev/null
  Time (mean ± σ):     11.377 s ±  0.088 s    [User: 14.202 s, System: 0.497 s]
  Range (min … max):   11.296 s … 11.599 s    10 runs

Summary
  gix archive -f zip /dev/null ran
    2.19 ± 0.09 times faster than git archive --format zip @ >/dev/null

File Sizes

✦ ❯ git archive -o git.tar @ && gix archive gix.tar && ls -l *.tar
 12:45:37 entries done 79.6k entries in 3.17s (25.1k entries/s)
 12:45:37 written done 1.4GB in 3.17s (432.9MB/s)
-rw-r--r--  1 byron  staff  1375395840 Aug  7 12:45 git.tar
-rw-r--r--  1 byron  staff  1372750848 Aug  7 12:45 gix.tar

tar.gz libflate

linux ( master) +369 -819 [!] took 8s
✦ ❯ git archive -o git.tar.gz @ && gix archive gix.tar.gz && ls -l *.tar.gz
 12:47:03 entries done 79.6k entries in 21.89s (3.6k entries/s)
 12:47:03 written done 291.3MB in 21.89s (13.3MB/s)
-rw-r--r--  1 byron  staff  223185415 Aug  7 12:46 git.tar.gz
-rw-r--r--  1 byron  staff  291320714 Aug  7 12:47 gix.tar.gz

tar.gz flate2

✦ ❯ gix archive gix.tar.gz && ls -l *.tar.gz
 15:47:13 entries done 79.6k entries in 11.74s (6.8k entries/s)
 15:47:13 written done 223.0MB in 11.74s (19.0MB/s)
-rw-r--r--  1 byron  staff  223185415 Aug  7 15:10 git.tar.gz
-rw-r--r--@ 1 byron  staff  222953349 Aug  7 15:47 gix.tar.gz

✦ ❯ git archive -o git.zip @ && gix archive gix.zip && ls -l *.zip
 12:51:27 entries done 79.6k entries in 11.79s (6.8k entries/s)
 12:51:27 written done 266.4MB in 11.79s (22.6MB/s)
-rw-r--r--  1 byron  staff  267527033 Aug  7 12:51 git.zip
-rw-r--r--  1 byron  staff  265453241 Aug  7 12:51 gix.zip

tar.gz flate2 with compression level 9

✦ ❯ time git archive -9 -o git.9.tar.gz @ && time gix archive -f tar-gz -l 9 gix.9.tar.gz && ls -l *.tar.gz
git archive -9 -o git.9.tar.gz @  45.54s user 1.02s system 94% cpu 49.222 total
 16:05:35 entries done 79.6k entries in 22.11s (3.6k entries/s)
 16:05:35 written done 219.2MB in 22.11s (9.9MB/s)
gix archive -f tar-gz -l 9 gix.9.tar.gz  24.50s user 0.49s system 112% cpu 22.143 total
-rw-r--r--  1 byron  staff  219175582 Aug  7 16:05 git.9.tar.gz
-rw-r--r--  1 byron  staff  223185415 Aug  7 15:10 git.tar.gz
-rw-r--r--  1 byron  staff  219219361 Aug  7 16:05 gix.9.tar.gz
-rw-r--r--@ 1 byron  staff  222953349 Aug  7 15:47 gix.tar.gz

tar.gz flate2 with compression level 1

✦ ❯ time git archive -1 -o git.1.tar.gz @ && time gix archive -f tar-gz -l 1 gix.1.tar.gz && ls -l *.tar.gz
git archive -1 -o git.1.tar.gz @  10.62s user 0.41s system 99% cpu 11.088 total
 16:31:38 entries done 79.6k entries in 4.22s (18.9k entries/s)
 16:31:38 written done 345.8MB in 4.22s (81.9MB/s)
gix archive -f tar-gz -l 1 gix.1.tar.gz  6.99s user 0.22s system 169% cpu 4.261 total
-rw-r--r--  1 byron  staff  282942920 Aug  7 16:31 git.1.tar.gz
-rw-r--r--  1 byron  staff  345797802 Aug  7 16:31 gix.1.tar.gz

Memory

✦ ❯ /usr/bin/time -lp git archive --format tar @ >/dev/null && /usr/bin/time -lp gix archive -f tar /dev/null
real 4.41
user 4.16
sys 0.23
          1245200384  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
               76460  page reclaims
                   7  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                   0  voluntary context switches
                 182  involuntary context switches
         27864098025  instructions retired
         14150963412  cycles elapsed
           159698944  peak memory footprint
 13:02:00 entries done 79.6k entries in 3.15s (25.2k entries/s)
 13:02:00 written done 1.4GB in 3.15s (435.2MB/s)
real 3.16
user 3.27
sys 0.22
          1063206912  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
               71040  page reclaims
                  30  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                   0  voluntary context switches
               40644  involuntary context switches
         22979191630  instructions retired
         11015589733  cycles elapsed
           154489536  peak memory footprint

with libflate

✦ ❯ /usr/bin/time -lp git archive --format tar.gz @ >/dev/null && /usr/bin/time -lp gix archive -f tar-gz /dev/null
real 26.10
user 25.70
sys 0.38
          1493630976  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
               91625  page reclaims
                   7  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                   0  voluntary context switches
                 157  involuntary context switches
        167416140948  instructions retired
         84052925355  cycles elapsed
           172724544  peak memory footprint
 13:09:54 entries done 79.6k entries in 21.50s (3.7k entries/s)
 13:09:54 written done 291.3MB in 21.50s (13.5MB/s)
real 21.53
user 24.20
sys 0.28
          1245036544  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
               81222  page reclaims
                 403  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                  13  voluntary context switches
                8480  involuntary context switches
        201695201211  instructions retired
         78325547595  cycles elapsed
           241390528  peak memory footprint

with flate2

✦ ❯ /usr/bin/time -lp gix archive -f tar-gz /dev/null
 15:45:13 entries done 79.6k entries in 11.45s (7.0k entries/s)
 15:45:13 written done 223.0MB in 11.45s (19.5MB/s)
real 11.46
user 14.40
sys 0.30
          1237368832  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
               78742  page reclaims
                  27  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                   2  voluntary context switches
               30526  involuntary context switches
        115032760252  instructions retired
         46646222913  cycles elapsed
           258069568  peak memory footprint

✦ ❯ /usr/bin/time -lp git archive --format zip @ >/dev/null && /usr/bin/time -lp gix archive -f zip /dev/null
real 25.10
user 22.88
sys 2.20
          1621786624  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
              100170  page reclaims
                   7  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                   0  voluntary context switches
                 428  involuntary context switches
        173958199176  instructions retired
         80702825962  cycles elapsed
           167628160  peak memory footprint
 13:11:37 entries done 79.6k entries in 11.36s (7.0k entries/s)
 13:11:37 written done 266.4MB in 11.36s (23.5MB/s)
real 11.39
user 14.22
sys 0.49
          1276805120  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
               80086  page reclaims
                 400  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                  13  voluntary context switches
               59638  involuntary context switches
        122272800725  instructions retired
         46388176051  cycles elapsed
           247453184  peak memory footprint

[Shnatsel]

When looking at tar.gz, the compression level is certainly completely different from what git uses, and unfortunately it's unclear to me how to configure this in the crate I am using.

Which crate is that?

[Byron]

I can't resist to try that right now and update the tables :D. Thanks for the hint!

[Byron]

Fantastic! Using flate2 fixes all the performance problems!

It also provides multiple gzip compression examples in the repository, including streaming support [..]

It's worth noting that the problem with tar.gz isn't the lack of streaming support for the gz part, but the lack of streaming support for the API offered by tar. tar doesn't seem to have a 'chunked' format and needs to know the size of the entry in advance. Without Seek support for the writer there seems to be no way to do it without reading the entry into memory first.

[cj-timothy-sampson]

is there a tracking bug for submodule support in this (I didn't see any)? And I assume once submodule support were added it would supplant https://github.com/Kentzo/git-archive-all?

[Byron]

I did check and found this entry in `crate-status indicating that it definitely isn't implemented yet.
There is also no tracking ticket and I have no plans to implement this anytime soon. However, I do agree that it would supplant anything else when implemented, it will be crazy fast :).

But here is the tracking ticket.

[cj-timothy-sampson]

I did check and found this entry in `crate-status indicating that it definitely isn't implemented yet.

Apologies, I didn't think to check there.

But #1182.

Thanks! (Also for gitoxide in general!)