Sparse index: integrate with `git update-index` #423

vdye · 2021-09-09T13:39:27Z

Changes

Tests to cover various update-index scenarios (focused on those that use the index), setting a baseline for expected behavior once sparse index is integrated.
Integrations for allowing update-index to work with a sparse index. In this case, these are entirely made up of:
- add prepare_repo_settings and set command_requires_full_index = 0
- ensure the full index if a file outside the sparse checkout definition is referenced
- return an error if the command attempts to directly modify a sparse directory entry
- new test to ensure the index isn't expanded when operating inside sparse cone

Note on modifying sparse directory entries: there may be a case here for adding a --sparse flag to allow update-index operations on sparse directory index entries, but it seems more prone to strange behavior/breaking things (e.g., manually removing the skip-worktree flag) than ls-files or other commands where that has been used. No functionality is reduced by disallowing changes to sparse directories either, since the command today only allows operations on files.

derrickstolee

To get it ready for what we need in microsoft/git, there are only some nits.

To get it ready for upstream submission, there might be some hard work to reorganize the commits. I'm not even sure if such a split is possible, but it might be worth trying.

t/t1092-sparse-checkout-compatibility.sh

builtin/update-index.c

t/t1092-sparse-checkout-compatibility.sh

builtin/update-index.c

Add explicit check for directory entry mode, throwing an error if found. Without the check, sparse directory cache entries could be added to any index (including non-sparse). In a sparse index, this would at least cause inconsistencies in the cache tree; for a non-sparse index, such an entry being present at all is a bug. Signed-off-by: Victoria Dye <vdye@github.com>

read-cache.c

t/t1092-sparse-checkout-compatibility.sh

derrickstolee

I have one question about the do_reupdate() change, and it might be answered quickly with a test update.

derrickstolee · 2021-09-13T14:35:03Z

builtin/update-index.c

+		/*
+		 * We can safely skip re-updating sparse directories because if there
+		 * were any changes to re-update inside of the sparse directory, it
+		 * would not be sparse.
+		 */
+		if (S_ISSPARSEDIR(ce->ce_mode) || ce_stage(ce) ||
+		    !ce_path_match(&the_index, ce, &pathspec, NULL))
 			continue;
+


Do you have tests for this re-update mode when the file actually exists outside of the sparse-checkout cone? For example, a user might manually write the file to disk. The change you are making here might be a reasonable choice, regardless, but it might also be a behavior change that we'd want to document with a tests.

I've pushed an update to the test that verifies update-index --again - it now explicitly tests:

file manually added and modified outside sparse definition (without skip-worktree disabled)

result: no-op (update-index --again only operates on staged changes in the index)

file manually added outside sparse definition (by disabling skip-worktree on that file) and modified

in all checkouts, file is re-updated

file manually removed after disabling skip-worktree

in all checkouts, file cannot be removed without --remove

t/t1092-sparse-checkout-compatibility.sh

In preparation of integrating 'git update-index' with the sparse index, introduce tests for a variety of 'git update-index' uses. Some (namely those in update-index add/remove`) are focused specifically on how 'git stash' uses 'git update-index' as a subcommand. Signed-off-by: Victoria Dye <vdye@github.com>

Enable usage of the sparse index with `update-index`. Most variations of `update-index` work without explicitly expanding the index (due to the implicit expansion performed when attempting to find an existing entry). However, in order to prevent manually creating a sparse directory cache entry (current behavior is to throw an error when presented with a directory using `--cacheinfo`), the index is expanded before the the entry is added. Signed-off-by: Victoria Dye <vdye@github.com>

Rearrange `add_index_entry_with_check` to allow `index_name_stage_pos` to expand the index _before_ attempting to invalidate the relevant cache tree path. This permits implicit index expansion when adding a cache entry outside the sparse checkout definition. Signed-off-by: Victoria Dye <vdye@github.com>

Instead of ensuring the full index, only need to prevent reupdating sparse directory entries to maintain expected behavior. Corresponding addition to index expansion test verifies the sparse index is not expanded. Signed-off-by: Victoria Dye <vdye@github.com>

derrickstolee

Thanks for adding the new test. I'm happy it didn't require changing your implementation (because sparse entries were being ignored, anyway).

One thing I forgot when talking about the sparse index is that we have a performance test: `t/perf/p2000-sparse-operations.sh`. This test wasn't helpful for commands like `git merge` that need a particular set of input, but work for more read-only operations. Here is a quick demonstration of how this performance test works so we could have a definitive measure of how your previous updates improved performance. To get these results, I ran the following command in `t/perf`: ``` ./run 4bcd533 f9255a5 f28fc01 b713582 -- p2000-sparse-operations.sh ``` The short-shas correspond to the merge commits for these PRs: * #410 * #421 * #417 * #419 The test takes a copy of the Git repository and creates several copies within a nested directory heirarchy. ``` Test 4bcd533 f9255a5 f28fc01 b713582 ------------------------------------------------------------------------------------------------------------------------------------------------- 2000.2: git status (full-v3) 0.19(0.15+0.05) 0.19(0.16+0.05) +0.0% 0.20(0.18+0.03) +5.3% 0.19(0.17+0.04) +0.0% 2000.3: git status (full-v4) 0.20(0.18+0.04) 0.19(0.15+0.06) -5.0% 0.21(0.18+0.05) +5.0% 0.18(0.18+0.02) -10.0% 2000.4: git status (sparse-v3) 0.04(0.04+0.04) 0.05(0.07+0.04) +25.0% 0.04(0.04+0.05) +0.0% 0.04(0.06+0.04) +0.0% 2000.5: git status (sparse-v4) 0.04(0.03+0.06) 0.04(0.05+0.05) +0.0% 0.05(0.05+0.04) +25.0% 0.05(0.06+0.04) +25.0% 2000.6: git add -A (full-v3) 0.36(0.29+0.05) 0.38(0.28+0.07) +5.6% 0.36(0.31+0.05) +0.0% 0.37(0.31+0.05) +2.8% 2000.7: git add -A (full-v4) 0.34(0.27+0.06) 0.34(0.29+0.05) +0.0% 0.34(0.29+0.04) +0.0% 0.35(0.28+0.06) +2.9% 2000.8: git add -A (sparse-v3) 0.06(0.07+0.04) 0.06(0.05+0.06) +0.0% 0.06(0.09+0.01) +0.0% 0.06(0.08+0.03) +0.0% 2000.9: git add -A (sparse-v4) 0.05(0.05+0.04) 0.05(0.05+0.07) +0.0% 0.05(0.04+0.06) +0.0% 0.06(0.06+0.05) +20.0% 2000.10: git add . (full-v3) 0.38(0.31+0.05) 0.37(0.29+0.06) -2.6% 0.37(0.30+0.07) -2.6% 0.37(0.29+0.06) -2.6% 2000.11: git add . (full-v4) 0.35(0.31+0.04) 0.35(0.29+0.07) +0.0% 0.35(0.29+0.05) +0.0% 0.34(0.29+0.06) -2.9% 2000.12: git add . (sparse-v3) 0.06(0.06+0.05) 0.06(0.05+0.06) +0.0% 0.06(0.07+0.05) +0.0% 0.06(0.09+0.03) +0.0% 2000.13: git add . (sparse-v4) 0.06(0.06+0.06) 0.06(0.07+0.04) +0.0% 0.05(0.06+0.05) -16.7% 0.05(0.05+0.07) -16.7% 2000.14: git commit -a -m A (full-v3) 0.48(0.37+0.08) 0.45(0.36+0.08) -6.2% 0.45(0.35+0.09) -6.2% 0.44(0.36+0.07) -8.3% 2000.15: git commit -a -m A (full-v4) 0.45(0.40+0.06) 0.43(0.34+0.07) -4.4% 0.45(0.37+0.06) +0.0% 0.42(0.36+0.05) -6.7% 2000.16: git commit -a -m A (sparse-v3) 0.05(0.05+0.06) 0.05(0.05+0.03) +0.0% 0.05(0.06+0.06) +0.0% 0.05(0.04+0.06) +0.0% 2000.17: git commit -a -m A (sparse-v4) 0.05(0.06+0.03) 0.05(0.06+0.04) +0.0% 0.06(0.07+0.05) +20.0% 0.05(0.04+0.06) +0.0% 2000.18: git checkout -f - (full-v3) 0.55(0.43+0.08) 0.54(0.46+0.05) -1.8% 0.55(0.46+0.07) +0.0% 0.54(0.40+0.10) -1.8% 2000.19: git checkout -f - (full-v4) 0.55(0.41+0.09) 0.50(0.40+0.09) -9.1% 0.51(0.46+0.05) -7.3% 0.51(0.44+0.06) -7.3% 2000.20: git checkout -f - (sparse-v3) 0.06(0.09+0.03) 0.06(0.08+0.03) +0.0% 0.06(0.06+0.05) +0.0% 0.07(0.09+0.03) +16.7% 2000.21: git checkout -f - (sparse-v4) 0.06(0.08+0.04) 0.05(0.07+0.05) -16.7% 0.05(0.07+0.04) -16.7% 0.06(0.09+0.03) +0.0% ``` All of the above were already integrated. ``` 2000.22: git reset (full-v3) 0.41(0.32+0.06) 0.40(0.31+0.06) -2.4% 0.41(0.33+0.05) +0.0% 0.42(0.34+0.04) +2.4% 2000.23: git reset (full-v4) 0.37(0.32+0.05) 0.35(0.30+0.05) -5.4% 0.37(0.30+0.05) +0.0% 0.35(0.31+0.03) -5.4% 2000.24: git reset (sparse-v3) 0.68(0.65+0.05) 0.55(0.52+0.04) -19.1% 0.04(0.05+0.04) -94.1% 0.04(0.05+0.04) -94.1% 2000.25: git reset (sparse-v4) 0.70(0.65+0.05) 0.54(0.50+0.06) -22.9% 0.04(0.07+0.01) -94.3% 0.03(0.05+0.05) -95.7% 2000.26: git reset --hard (full-v3) 0.54(0.43+0.07) 0.53(0.43+0.06) -1.9% 0.55(0.46+0.05) +1.9% 0.55(0.44+0.06) +1.9% 2000.27: git reset --hard (full-v4) 0.50(0.45+0.03) 0.50(0.43+0.05) +0.0% 0.49(0.41+0.06) -2.0% 0.50(0.42+0.05) +0.0% 2000.28: git reset --hard (sparse-v3) 0.83(0.76+0.06) 0.68(0.62+0.05) -18.1% 0.07(0.05+0.02) -91.6% 0.07(0.05+0.02) -91.6% 2000.29: git reset --hard (sparse-v4) 0.80(0.75+0.05) 0.69(0.62+0.06) -13.8% 0.07(0.04+0.02) -91.2% 0.07(0.04+0.03) -91.2% ``` As expected, `git reset [--hard]` improves with the sparse index integration, but remains constant across the full index case. ``` 2000.30: git update-index --add --remove (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.01+0.01) +0.0% 2000.31: git update-index --add --remove (full-v4) 0.03(0.02+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.03+0.00) +0.0% 0.03(0.02+0.01) +0.0% 2000.32: git update-index --add --remove (sparse-v3) 0.57(0.54+0.02) 0.43(0.42+0.00) -24.6% 0.44(0.41+0.03) -22.8% 0.44(0.42+0.01) -22.8% 2000.33: git update-index --add --remove (sparse-v4) 0.56(0.52+0.04) 0.43(0.42+0.01) -23.2% 0.44(0.42+0.02) -21.4% 0.42(0.41+0.01) -25.0% ``` These do not change significantly because #423 is not merged. ``` 2000.34: git diff (full-v3) 0.07(0.05+0.03) 0.06(0.05+0.03) -14.3% 0.07(0.05+0.03) +0.0% 0.06(0.05+0.03) -14.3% 2000.35: git diff (full-v4) 0.06(0.05+0.03) 0.06(0.05+0.02) +0.0% 0.06(0.05+0.02) +0.0% 0.06(0.06+0.02) +0.0% 2000.36: git diff (sparse-v3) 0.25(0.23+0.03) 0.17(0.17+0.02) -32.0% 0.18(0.18+0.02) -28.0% 0.01(0.03+0.03) -96.0% 2000.37: git diff (sparse-v4) 0.25(0.22+0.05) 0.16(0.16+0.01) -36.0% 0.18(0.15+0.04) -28.0% 0.01(0.04+0.02) -96.0% 2000.38: git diff --staged (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.02+0.00) +0.0% 2000.39: git diff --staged (full-v4) 0.04(0.03+0.01) 0.03(0.02+0.01) -25.0% 0.03(0.03+0.00) -25.0% 0.03(0.03+0.00) -25.0% 2000.40: git diff --staged (sparse-v3) 0.21(0.19+0.01) 0.15(0.13+0.01) -28.6% 0.15(0.14+0.01) -28.6% 0.01(0.01+0.00) -95.2% 2000.41: git diff --staged (sparse-v4) 0.22(0.21+0.01) 0.14(0.11+0.03) -36.4% 0.15(0.13+0.02) -31.8% 0.01(0.01+0.00) -95.5% ``` The `git diff` improvements are measurable. ``` 2000.42: git sparse-checkout reapply (full-v3) 0.63(0.54+0.05) 0.56(0.48+0.04) -11.1% 0.57(0.48+0.03) -9.5% 0.59(0.48+0.05) -6.3% 2000.43: git sparse-checkout reapply (full-v4) 0.60(0.54+0.02) 0.51(0.46+0.03) -15.0% 0.54(0.48+0.02) -10.0% 0.50(0.44+0.04) -16.7% 2000.44: git sparse-checkout reapply (sparse-v3) 0.91(0.86+0.05) 0.05(0.05+0.00) -94.5% 0.06(0.05+0.01) -93.4% 0.06(0.06+0.00) -93.4% 2000.45: git sparse-checkout reapply (sparse-v4) 0.92(0.88+0.04) 0.05(0.05+0.00) -94.6% 0.05(0.05+0.01) -94.6% 0.05(0.04+0.01) -94.6% ``` Finally, the `git sparse-checkout` measurements are also present. This test script is particularly valuable when contributing changes upstream. It can be good to start by adding the lines to the performance test in an early commit, then demonstrating the performance change by copying the necessary lines from the output table into your commit message.

Sparse index: integrate with `git update-index`

One thing I forgot when talking about the sparse index is that we have a performance test: `t/perf/p2000-sparse-operations.sh`. This test wasn't helpful for commands like `git merge` that need a particular set of input, but work for more read-only operations. Here is a quick demonstration of how this performance test works so we could have a definitive measure of how your previous updates improved performance. To get these results, I ran the following command in `t/perf`: ``` ./run 4bcd533 f9255a5 f28fc01 b713582 -- p2000-sparse-operations.sh ``` The short-shas correspond to the merge commits for these PRs: * #410 * #421 * #417 * #419 The test takes a copy of the Git repository and creates several copies within a nested directory heirarchy. ``` Test 4bcd533 f9255a5 f28fc01 b713582 ------------------------------------------------------------------------------------------------------------------------------------------------- 2000.2: git status (full-v3) 0.19(0.15+0.05) 0.19(0.16+0.05) +0.0% 0.20(0.18+0.03) +5.3% 0.19(0.17+0.04) +0.0% 2000.3: git status (full-v4) 0.20(0.18+0.04) 0.19(0.15+0.06) -5.0% 0.21(0.18+0.05) +5.0% 0.18(0.18+0.02) -10.0% 2000.4: git status (sparse-v3) 0.04(0.04+0.04) 0.05(0.07+0.04) +25.0% 0.04(0.04+0.05) +0.0% 0.04(0.06+0.04) +0.0% 2000.5: git status (sparse-v4) 0.04(0.03+0.06) 0.04(0.05+0.05) +0.0% 0.05(0.05+0.04) +25.0% 0.05(0.06+0.04) +25.0% 2000.6: git add -A (full-v3) 0.36(0.29+0.05) 0.38(0.28+0.07) +5.6% 0.36(0.31+0.05) +0.0% 0.37(0.31+0.05) +2.8% 2000.7: git add -A (full-v4) 0.34(0.27+0.06) 0.34(0.29+0.05) +0.0% 0.34(0.29+0.04) +0.0% 0.35(0.28+0.06) +2.9% 2000.8: git add -A (sparse-v3) 0.06(0.07+0.04) 0.06(0.05+0.06) +0.0% 0.06(0.09+0.01) +0.0% 0.06(0.08+0.03) +0.0% 2000.9: git add -A (sparse-v4) 0.05(0.05+0.04) 0.05(0.05+0.07) +0.0% 0.05(0.04+0.06) +0.0% 0.06(0.06+0.05) +20.0% 2000.10: git add . (full-v3) 0.38(0.31+0.05) 0.37(0.29+0.06) -2.6% 0.37(0.30+0.07) -2.6% 0.37(0.29+0.06) -2.6% 2000.11: git add . (full-v4) 0.35(0.31+0.04) 0.35(0.29+0.07) +0.0% 0.35(0.29+0.05) +0.0% 0.34(0.29+0.06) -2.9% 2000.12: git add . (sparse-v3) 0.06(0.06+0.05) 0.06(0.05+0.06) +0.0% 0.06(0.07+0.05) +0.0% 0.06(0.09+0.03) +0.0% 2000.13: git add . (sparse-v4) 0.06(0.06+0.06) 0.06(0.07+0.04) +0.0% 0.05(0.06+0.05) -16.7% 0.05(0.05+0.07) -16.7% 2000.14: git commit -a -m A (full-v3) 0.48(0.37+0.08) 0.45(0.36+0.08) -6.2% 0.45(0.35+0.09) -6.2% 0.44(0.36+0.07) -8.3% 2000.15: git commit -a -m A (full-v4) 0.45(0.40+0.06) 0.43(0.34+0.07) -4.4% 0.45(0.37+0.06) +0.0% 0.42(0.36+0.05) -6.7% 2000.16: git commit -a -m A (sparse-v3) 0.05(0.05+0.06) 0.05(0.05+0.03) +0.0% 0.05(0.06+0.06) +0.0% 0.05(0.04+0.06) +0.0% 2000.17: git commit -a -m A (sparse-v4) 0.05(0.06+0.03) 0.05(0.06+0.04) +0.0% 0.06(0.07+0.05) +20.0% 0.05(0.04+0.06) +0.0% 2000.18: git checkout -f - (full-v3) 0.55(0.43+0.08) 0.54(0.46+0.05) -1.8% 0.55(0.46+0.07) +0.0% 0.54(0.40+0.10) -1.8% 2000.19: git checkout -f - (full-v4) 0.55(0.41+0.09) 0.50(0.40+0.09) -9.1% 0.51(0.46+0.05) -7.3% 0.51(0.44+0.06) -7.3% 2000.20: git checkout -f - (sparse-v3) 0.06(0.09+0.03) 0.06(0.08+0.03) +0.0% 0.06(0.06+0.05) +0.0% 0.07(0.09+0.03) +16.7% 2000.21: git checkout -f - (sparse-v4) 0.06(0.08+0.04) 0.05(0.07+0.05) -16.7% 0.05(0.07+0.04) -16.7% 0.06(0.09+0.03) +0.0% ``` All of the above were already integrated. ``` 2000.22: git reset (full-v3) 0.41(0.32+0.06) 0.40(0.31+0.06) -2.4% 0.41(0.33+0.05) +0.0% 0.42(0.34+0.04) +2.4% 2000.23: git reset (full-v4) 0.37(0.32+0.05) 0.35(0.30+0.05) -5.4% 0.37(0.30+0.05) +0.0% 0.35(0.31+0.03) -5.4% 2000.24: git reset (sparse-v3) 0.68(0.65+0.05) 0.55(0.52+0.04) -19.1% 0.04(0.05+0.04) -94.1% 0.04(0.05+0.04) -94.1% 2000.25: git reset (sparse-v4) 0.70(0.65+0.05) 0.54(0.50+0.06) -22.9% 0.04(0.07+0.01) -94.3% 0.03(0.05+0.05) -95.7% 2000.26: git reset --hard (full-v3) 0.54(0.43+0.07) 0.53(0.43+0.06) -1.9% 0.55(0.46+0.05) +1.9% 0.55(0.44+0.06) +1.9% 2000.27: git reset --hard (full-v4) 0.50(0.45+0.03) 0.50(0.43+0.05) +0.0% 0.49(0.41+0.06) -2.0% 0.50(0.42+0.05) +0.0% 2000.28: git reset --hard (sparse-v3) 0.83(0.76+0.06) 0.68(0.62+0.05) -18.1% 0.07(0.05+0.02) -91.6% 0.07(0.05+0.02) -91.6% 2000.29: git reset --hard (sparse-v4) 0.80(0.75+0.05) 0.69(0.62+0.06) -13.8% 0.07(0.04+0.02) -91.2% 0.07(0.04+0.03) -91.2% ``` As expected, `git reset [--hard]` improves with the sparse index integration, but remains constant across the full index case. ``` 2000.30: git update-index --add --remove (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.01+0.01) +0.0% 2000.31: git update-index --add --remove (full-v4) 0.03(0.02+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.03+0.00) +0.0% 0.03(0.02+0.01) +0.0% 2000.32: git update-index --add --remove (sparse-v3) 0.57(0.54+0.02) 0.43(0.42+0.00) -24.6% 0.44(0.41+0.03) -22.8% 0.44(0.42+0.01) -22.8% 2000.33: git update-index --add --remove (sparse-v4) 0.56(0.52+0.04) 0.43(0.42+0.01) -23.2% 0.44(0.42+0.02) -21.4% 0.42(0.41+0.01) -25.0% ``` These do not change significantly because #423 is not merged. ``` 2000.34: git diff (full-v3) 0.07(0.05+0.03) 0.06(0.05+0.03) -14.3% 0.07(0.05+0.03) +0.0% 0.06(0.05+0.03) -14.3% 2000.35: git diff (full-v4) 0.06(0.05+0.03) 0.06(0.05+0.02) +0.0% 0.06(0.05+0.02) +0.0% 0.06(0.06+0.02) +0.0% 2000.36: git diff (sparse-v3) 0.25(0.23+0.03) 0.17(0.17+0.02) -32.0% 0.18(0.18+0.02) -28.0% 0.01(0.03+0.03) -96.0% 2000.37: git diff (sparse-v4) 0.25(0.22+0.05) 0.16(0.16+0.01) -36.0% 0.18(0.15+0.04) -28.0% 0.01(0.04+0.02) -96.0% 2000.38: git diff --staged (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.02+0.00) +0.0% 2000.39: git diff --staged (full-v4) 0.04(0.03+0.01) 0.03(0.02+0.01) -25.0% 0.03(0.03+0.00) -25.0% 0.03(0.03+0.00) -25.0% 2000.40: git diff --staged (sparse-v3) 0.21(0.19+0.01) 0.15(0.13+0.01) -28.6% 0.15(0.14+0.01) -28.6% 0.01(0.01+0.00) -95.2% 2000.41: git diff --staged (sparse-v4) 0.22(0.21+0.01) 0.14(0.11+0.03) -36.4% 0.15(0.13+0.02) -31.8% 0.01(0.01+0.00) -95.5% ``` The `git diff` improvements are measurable. ``` 2000.42: git sparse-checkout reapply (full-v3) 0.63(0.54+0.05) 0.56(0.48+0.04) -11.1% 0.57(0.48+0.03) -9.5% 0.59(0.48+0.05) -6.3% 2000.43: git sparse-checkout reapply (full-v4) 0.60(0.54+0.02) 0.51(0.46+0.03) -15.0% 0.54(0.48+0.02) -10.0% 0.50(0.44+0.04) -16.7% 2000.44: git sparse-checkout reapply (sparse-v3) 0.91(0.86+0.05) 0.05(0.05+0.00) -94.5% 0.06(0.05+0.01) -93.4% 0.06(0.06+0.00) -93.4% 2000.45: git sparse-checkout reapply (sparse-v4) 0.92(0.88+0.04) 0.05(0.05+0.00) -94.6% 0.05(0.05+0.01) -94.6% 0.05(0.04+0.01) -94.6% ``` Finally, the `git sparse-checkout` measurements are also present. This test script is particularly valuable when contributing changes upstream. It can be good to start by adding the lines to the performance test in an early commit, then demonstrating the performance change by copying the necessary lines from the output table into your commit message.

Sparse index: integrate with `git update-index`

One thing I forgot when talking about the sparse index is that we have a performance test: `t/perf/p2000-sparse-operations.sh`. This test wasn't helpful for commands like `git merge` that need a particular set of input, but work for more read-only operations. Here is a quick demonstration of how this performance test works so we could have a definitive measure of how your previous updates improved performance. To get these results, I ran the following command in `t/perf`: ``` ./run 4bcd533 f9255a5 f28fc01 b713582 -- p2000-sparse-operations.sh ``` The short-shas correspond to the merge commits for these PRs: * #410 * #421 * #417 * #419 The test takes a copy of the Git repository and creates several copies within a nested directory heirarchy. ``` Test 4bcd533 f9255a5 f28fc01 b713582 ------------------------------------------------------------------------------------------------------------------------------------------------- 2000.2: git status (full-v3) 0.19(0.15+0.05) 0.19(0.16+0.05) +0.0% 0.20(0.18+0.03) +5.3% 0.19(0.17+0.04) +0.0% 2000.3: git status (full-v4) 0.20(0.18+0.04) 0.19(0.15+0.06) -5.0% 0.21(0.18+0.05) +5.0% 0.18(0.18+0.02) -10.0% 2000.4: git status (sparse-v3) 0.04(0.04+0.04) 0.05(0.07+0.04) +25.0% 0.04(0.04+0.05) +0.0% 0.04(0.06+0.04) +0.0% 2000.5: git status (sparse-v4) 0.04(0.03+0.06) 0.04(0.05+0.05) +0.0% 0.05(0.05+0.04) +25.0% 0.05(0.06+0.04) +25.0% 2000.6: git add -A (full-v3) 0.36(0.29+0.05) 0.38(0.28+0.07) +5.6% 0.36(0.31+0.05) +0.0% 0.37(0.31+0.05) +2.8% 2000.7: git add -A (full-v4) 0.34(0.27+0.06) 0.34(0.29+0.05) +0.0% 0.34(0.29+0.04) +0.0% 0.35(0.28+0.06) +2.9% 2000.8: git add -A (sparse-v3) 0.06(0.07+0.04) 0.06(0.05+0.06) +0.0% 0.06(0.09+0.01) +0.0% 0.06(0.08+0.03) +0.0% 2000.9: git add -A (sparse-v4) 0.05(0.05+0.04) 0.05(0.05+0.07) +0.0% 0.05(0.04+0.06) +0.0% 0.06(0.06+0.05) +20.0% 2000.10: git add . (full-v3) 0.38(0.31+0.05) 0.37(0.29+0.06) -2.6% 0.37(0.30+0.07) -2.6% 0.37(0.29+0.06) -2.6% 2000.11: git add . (full-v4) 0.35(0.31+0.04) 0.35(0.29+0.07) +0.0% 0.35(0.29+0.05) +0.0% 0.34(0.29+0.06) -2.9% 2000.12: git add . (sparse-v3) 0.06(0.06+0.05) 0.06(0.05+0.06) +0.0% 0.06(0.07+0.05) +0.0% 0.06(0.09+0.03) +0.0% 2000.13: git add . (sparse-v4) 0.06(0.06+0.06) 0.06(0.07+0.04) +0.0% 0.05(0.06+0.05) -16.7% 0.05(0.05+0.07) -16.7% 2000.14: git commit -a -m A (full-v3) 0.48(0.37+0.08) 0.45(0.36+0.08) -6.2% 0.45(0.35+0.09) -6.2% 0.44(0.36+0.07) -8.3% 2000.15: git commit -a -m A (full-v4) 0.45(0.40+0.06) 0.43(0.34+0.07) -4.4% 0.45(0.37+0.06) +0.0% 0.42(0.36+0.05) -6.7% 2000.16: git commit -a -m A (sparse-v3) 0.05(0.05+0.06) 0.05(0.05+0.03) +0.0% 0.05(0.06+0.06) +0.0% 0.05(0.04+0.06) +0.0% 2000.17: git commit -a -m A (sparse-v4) 0.05(0.06+0.03) 0.05(0.06+0.04) +0.0% 0.06(0.07+0.05) +20.0% 0.05(0.04+0.06) +0.0% 2000.18: git checkout -f - (full-v3) 0.55(0.43+0.08) 0.54(0.46+0.05) -1.8% 0.55(0.46+0.07) +0.0% 0.54(0.40+0.10) -1.8% 2000.19: git checkout -f - (full-v4) 0.55(0.41+0.09) 0.50(0.40+0.09) -9.1% 0.51(0.46+0.05) -7.3% 0.51(0.44+0.06) -7.3% 2000.20: git checkout -f - (sparse-v3) 0.06(0.09+0.03) 0.06(0.08+0.03) +0.0% 0.06(0.06+0.05) +0.0% 0.07(0.09+0.03) +16.7% 2000.21: git checkout -f - (sparse-v4) 0.06(0.08+0.04) 0.05(0.07+0.05) -16.7% 0.05(0.07+0.04) -16.7% 0.06(0.09+0.03) +0.0% ``` All of the above were already integrated. ``` 2000.22: git reset (full-v3) 0.41(0.32+0.06) 0.40(0.31+0.06) -2.4% 0.41(0.33+0.05) +0.0% 0.42(0.34+0.04) +2.4% 2000.23: git reset (full-v4) 0.37(0.32+0.05) 0.35(0.30+0.05) -5.4% 0.37(0.30+0.05) +0.0% 0.35(0.31+0.03) -5.4% 2000.24: git reset (sparse-v3) 0.68(0.65+0.05) 0.55(0.52+0.04) -19.1% 0.04(0.05+0.04) -94.1% 0.04(0.05+0.04) -94.1% 2000.25: git reset (sparse-v4) 0.70(0.65+0.05) 0.54(0.50+0.06) -22.9% 0.04(0.07+0.01) -94.3% 0.03(0.05+0.05) -95.7% 2000.26: git reset --hard (full-v3) 0.54(0.43+0.07) 0.53(0.43+0.06) -1.9% 0.55(0.46+0.05) +1.9% 0.55(0.44+0.06) +1.9% 2000.27: git reset --hard (full-v4) 0.50(0.45+0.03) 0.50(0.43+0.05) +0.0% 0.49(0.41+0.06) -2.0% 0.50(0.42+0.05) +0.0% 2000.28: git reset --hard (sparse-v3) 0.83(0.76+0.06) 0.68(0.62+0.05) -18.1% 0.07(0.05+0.02) -91.6% 0.07(0.05+0.02) -91.6% 2000.29: git reset --hard (sparse-v4) 0.80(0.75+0.05) 0.69(0.62+0.06) -13.8% 0.07(0.04+0.02) -91.2% 0.07(0.04+0.03) -91.2% ``` As expected, `git reset [--hard]` improves with the sparse index integration, but remains constant across the full index case. ``` 2000.30: git update-index --add --remove (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.01+0.01) +0.0% 2000.31: git update-index --add --remove (full-v4) 0.03(0.02+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.03+0.00) +0.0% 0.03(0.02+0.01) +0.0% 2000.32: git update-index --add --remove (sparse-v3) 0.57(0.54+0.02) 0.43(0.42+0.00) -24.6% 0.44(0.41+0.03) -22.8% 0.44(0.42+0.01) -22.8% 2000.33: git update-index --add --remove (sparse-v4) 0.56(0.52+0.04) 0.43(0.42+0.01) -23.2% 0.44(0.42+0.02) -21.4% 0.42(0.41+0.01) -25.0% ``` These do not change significantly because #423 is not merged. ``` 2000.34: git diff (full-v3) 0.07(0.05+0.03) 0.06(0.05+0.03) -14.3% 0.07(0.05+0.03) +0.0% 0.06(0.05+0.03) -14.3% 2000.35: git diff (full-v4) 0.06(0.05+0.03) 0.06(0.05+0.02) +0.0% 0.06(0.05+0.02) +0.0% 0.06(0.06+0.02) +0.0% 2000.36: git diff (sparse-v3) 0.25(0.23+0.03) 0.17(0.17+0.02) -32.0% 0.18(0.18+0.02) -28.0% 0.01(0.03+0.03) -96.0% 2000.37: git diff (sparse-v4) 0.25(0.22+0.05) 0.16(0.16+0.01) -36.0% 0.18(0.15+0.04) -28.0% 0.01(0.04+0.02) -96.0% 2000.38: git diff --staged (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.02+0.00) +0.0% 2000.39: git diff --staged (full-v4) 0.04(0.03+0.01) 0.03(0.02+0.01) -25.0% 0.03(0.03+0.00) -25.0% 0.03(0.03+0.00) -25.0% 2000.40: git diff --staged (sparse-v3) 0.21(0.19+0.01) 0.15(0.13+0.01) -28.6% 0.15(0.14+0.01) -28.6% 0.01(0.01+0.00) -95.2% 2000.41: git diff --staged (sparse-v4) 0.22(0.21+0.01) 0.14(0.11+0.03) -36.4% 0.15(0.13+0.02) -31.8% 0.01(0.01+0.00) -95.5% ``` The `git diff` improvements are measurable. ``` 2000.42: git sparse-checkout reapply (full-v3) 0.63(0.54+0.05) 0.56(0.48+0.04) -11.1% 0.57(0.48+0.03) -9.5% 0.59(0.48+0.05) -6.3% 2000.43: git sparse-checkout reapply (full-v4) 0.60(0.54+0.02) 0.51(0.46+0.03) -15.0% 0.54(0.48+0.02) -10.0% 0.50(0.44+0.04) -16.7% 2000.44: git sparse-checkout reapply (sparse-v3) 0.91(0.86+0.05) 0.05(0.05+0.00) -94.5% 0.06(0.05+0.01) -93.4% 0.06(0.06+0.00) -93.4% 2000.45: git sparse-checkout reapply (sparse-v4) 0.92(0.88+0.04) 0.05(0.05+0.00) -94.6% 0.05(0.05+0.01) -94.6% 0.05(0.04+0.01) -94.6% ``` Finally, the `git sparse-checkout` measurements are also present. This test script is particularly valuable when contributing changes upstream. It can be good to start by adding the lines to the performance test in an early commit, then demonstrating the performance change by copying the necessary lines from the output table into your commit message.

Sparse index: integrate with `git update-index`

One thing I forgot when talking about the sparse index is that we have a performance test: `t/perf/p2000-sparse-operations.sh`. This test wasn't helpful for commands like `git merge` that need a particular set of input, but work for more read-only operations. Here is a quick demonstration of how this performance test works so we could have a definitive measure of how your previous updates improved performance. To get these results, I ran the following command in `t/perf`: ``` ./run 4bcd533 f9255a5 f28fc01 b713582 -- p2000-sparse-operations.sh ``` The short-shas correspond to the merge commits for these PRs: * #410 * #421 * #417 * #419 The test takes a copy of the Git repository and creates several copies within a nested directory heirarchy. ``` Test 4bcd533 f9255a5 f28fc01 b713582 ------------------------------------------------------------------------------------------------------------------------------------------------- 2000.2: git status (full-v3) 0.19(0.15+0.05) 0.19(0.16+0.05) +0.0% 0.20(0.18+0.03) +5.3% 0.19(0.17+0.04) +0.0% 2000.3: git status (full-v4) 0.20(0.18+0.04) 0.19(0.15+0.06) -5.0% 0.21(0.18+0.05) +5.0% 0.18(0.18+0.02) -10.0% 2000.4: git status (sparse-v3) 0.04(0.04+0.04) 0.05(0.07+0.04) +25.0% 0.04(0.04+0.05) +0.0% 0.04(0.06+0.04) +0.0% 2000.5: git status (sparse-v4) 0.04(0.03+0.06) 0.04(0.05+0.05) +0.0% 0.05(0.05+0.04) +25.0% 0.05(0.06+0.04) +25.0% 2000.6: git add -A (full-v3) 0.36(0.29+0.05) 0.38(0.28+0.07) +5.6% 0.36(0.31+0.05) +0.0% 0.37(0.31+0.05) +2.8% 2000.7: git add -A (full-v4) 0.34(0.27+0.06) 0.34(0.29+0.05) +0.0% 0.34(0.29+0.04) +0.0% 0.35(0.28+0.06) +2.9% 2000.8: git add -A (sparse-v3) 0.06(0.07+0.04) 0.06(0.05+0.06) +0.0% 0.06(0.09+0.01) +0.0% 0.06(0.08+0.03) +0.0% 2000.9: git add -A (sparse-v4) 0.05(0.05+0.04) 0.05(0.05+0.07) +0.0% 0.05(0.04+0.06) +0.0% 0.06(0.06+0.05) +20.0% 2000.10: git add . (full-v3) 0.38(0.31+0.05) 0.37(0.29+0.06) -2.6% 0.37(0.30+0.07) -2.6% 0.37(0.29+0.06) -2.6% 2000.11: git add . (full-v4) 0.35(0.31+0.04) 0.35(0.29+0.07) +0.0% 0.35(0.29+0.05) +0.0% 0.34(0.29+0.06) -2.9% 2000.12: git add . (sparse-v3) 0.06(0.06+0.05) 0.06(0.05+0.06) +0.0% 0.06(0.07+0.05) +0.0% 0.06(0.09+0.03) +0.0% 2000.13: git add . (sparse-v4) 0.06(0.06+0.06) 0.06(0.07+0.04) +0.0% 0.05(0.06+0.05) -16.7% 0.05(0.05+0.07) -16.7% 2000.14: git commit -a -m A (full-v3) 0.48(0.37+0.08) 0.45(0.36+0.08) -6.2% 0.45(0.35+0.09) -6.2% 0.44(0.36+0.07) -8.3% 2000.15: git commit -a -m A (full-v4) 0.45(0.40+0.06) 0.43(0.34+0.07) -4.4% 0.45(0.37+0.06) +0.0% 0.42(0.36+0.05) -6.7% 2000.16: git commit -a -m A (sparse-v3) 0.05(0.05+0.06) 0.05(0.05+0.03) +0.0% 0.05(0.06+0.06) +0.0% 0.05(0.04+0.06) +0.0% 2000.17: git commit -a -m A (sparse-v4) 0.05(0.06+0.03) 0.05(0.06+0.04) +0.0% 0.06(0.07+0.05) +20.0% 0.05(0.04+0.06) +0.0% 2000.18: git checkout -f - (full-v3) 0.55(0.43+0.08) 0.54(0.46+0.05) -1.8% 0.55(0.46+0.07) +0.0% 0.54(0.40+0.10) -1.8% 2000.19: git checkout -f - (full-v4) 0.55(0.41+0.09) 0.50(0.40+0.09) -9.1% 0.51(0.46+0.05) -7.3% 0.51(0.44+0.06) -7.3% 2000.20: git checkout -f - (sparse-v3) 0.06(0.09+0.03) 0.06(0.08+0.03) +0.0% 0.06(0.06+0.05) +0.0% 0.07(0.09+0.03) +16.7% 2000.21: git checkout -f - (sparse-v4) 0.06(0.08+0.04) 0.05(0.07+0.05) -16.7% 0.05(0.07+0.04) -16.7% 0.06(0.09+0.03) +0.0% ``` All of the above were already integrated. ``` 2000.22: git reset (full-v3) 0.41(0.32+0.06) 0.40(0.31+0.06) -2.4% 0.41(0.33+0.05) +0.0% 0.42(0.34+0.04) +2.4% 2000.23: git reset (full-v4) 0.37(0.32+0.05) 0.35(0.30+0.05) -5.4% 0.37(0.30+0.05) +0.0% 0.35(0.31+0.03) -5.4% 2000.24: git reset (sparse-v3) 0.68(0.65+0.05) 0.55(0.52+0.04) -19.1% 0.04(0.05+0.04) -94.1% 0.04(0.05+0.04) -94.1% 2000.25: git reset (sparse-v4) 0.70(0.65+0.05) 0.54(0.50+0.06) -22.9% 0.04(0.07+0.01) -94.3% 0.03(0.05+0.05) -95.7% 2000.26: git reset --hard (full-v3) 0.54(0.43+0.07) 0.53(0.43+0.06) -1.9% 0.55(0.46+0.05) +1.9% 0.55(0.44+0.06) +1.9% 2000.27: git reset --hard (full-v4) 0.50(0.45+0.03) 0.50(0.43+0.05) +0.0% 0.49(0.41+0.06) -2.0% 0.50(0.42+0.05) +0.0% 2000.28: git reset --hard (sparse-v3) 0.83(0.76+0.06) 0.68(0.62+0.05) -18.1% 0.07(0.05+0.02) -91.6% 0.07(0.05+0.02) -91.6% 2000.29: git reset --hard (sparse-v4) 0.80(0.75+0.05) 0.69(0.62+0.06) -13.8% 0.07(0.04+0.02) -91.2% 0.07(0.04+0.03) -91.2% ``` As expected, `git reset [--hard]` improves with the sparse index integration, but remains constant across the full index case. ``` 2000.30: git update-index --add --remove (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.01+0.01) +0.0% 2000.31: git update-index --add --remove (full-v4) 0.03(0.02+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.03+0.00) +0.0% 0.03(0.02+0.01) +0.0% 2000.32: git update-index --add --remove (sparse-v3) 0.57(0.54+0.02) 0.43(0.42+0.00) -24.6% 0.44(0.41+0.03) -22.8% 0.44(0.42+0.01) -22.8% 2000.33: git update-index --add --remove (sparse-v4) 0.56(0.52+0.04) 0.43(0.42+0.01) -23.2% 0.44(0.42+0.02) -21.4% 0.42(0.41+0.01) -25.0% ``` These do not change significantly because #423 is not merged. ``` 2000.34: git diff (full-v3) 0.07(0.05+0.03) 0.06(0.05+0.03) -14.3% 0.07(0.05+0.03) +0.0% 0.06(0.05+0.03) -14.3% 2000.35: git diff (full-v4) 0.06(0.05+0.03) 0.06(0.05+0.02) +0.0% 0.06(0.05+0.02) +0.0% 0.06(0.06+0.02) +0.0% 2000.36: git diff (sparse-v3) 0.25(0.23+0.03) 0.17(0.17+0.02) -32.0% 0.18(0.18+0.02) -28.0% 0.01(0.03+0.03) -96.0% 2000.37: git diff (sparse-v4) 0.25(0.22+0.05) 0.16(0.16+0.01) -36.0% 0.18(0.15+0.04) -28.0% 0.01(0.04+0.02) -96.0% 2000.38: git diff --staged (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.02+0.00) +0.0% 2000.39: git diff --staged (full-v4) 0.04(0.03+0.01) 0.03(0.02+0.01) -25.0% 0.03(0.03+0.00) -25.0% 0.03(0.03+0.00) -25.0% 2000.40: git diff --staged (sparse-v3) 0.21(0.19+0.01) 0.15(0.13+0.01) -28.6% 0.15(0.14+0.01) -28.6% 0.01(0.01+0.00) -95.2% 2000.41: git diff --staged (sparse-v4) 0.22(0.21+0.01) 0.14(0.11+0.03) -36.4% 0.15(0.13+0.02) -31.8% 0.01(0.01+0.00) -95.5% ``` The `git diff` improvements are measurable. ``` 2000.42: git sparse-checkout reapply (full-v3) 0.63(0.54+0.05) 0.56(0.48+0.04) -11.1% 0.57(0.48+0.03) -9.5% 0.59(0.48+0.05) -6.3% 2000.43: git sparse-checkout reapply (full-v4) 0.60(0.54+0.02) 0.51(0.46+0.03) -15.0% 0.54(0.48+0.02) -10.0% 0.50(0.44+0.04) -16.7% 2000.44: git sparse-checkout reapply (sparse-v3) 0.91(0.86+0.05) 0.05(0.05+0.00) -94.5% 0.06(0.05+0.01) -93.4% 0.06(0.06+0.00) -93.4% 2000.45: git sparse-checkout reapply (sparse-v4) 0.92(0.88+0.04) 0.05(0.05+0.00) -94.6% 0.05(0.05+0.01) -94.6% 0.05(0.04+0.01) -94.6% ``` Finally, the `git sparse-checkout` measurements are also present. This test script is particularly valuable when contributing changes upstream. It can be good to start by adding the lines to the performance test in an early commit, then demonstrating the performance change by copying the necessary lines from the output table into your commit message.

Sparse index: integrate with `git update-index`

One thing I forgot when talking about the sparse index is that we have a performance test: `t/perf/p2000-sparse-operations.sh`. This test wasn't helpful for commands like `git merge` that need a particular set of input, but work for more read-only operations. Here is a quick demonstration of how this performance test works so we could have a definitive measure of how your previous updates improved performance. To get these results, I ran the following command in `t/perf`: ``` ./run 4bcd533 f9255a5 f28fc01 b713582 -- p2000-sparse-operations.sh ``` The short-shas correspond to the merge commits for these PRs: * #410 * #421 * #417 * #419 The test takes a copy of the Git repository and creates several copies within a nested directory heirarchy. ``` Test 4bcd533 f9255a5 f28fc01 b713582 ------------------------------------------------------------------------------------------------------------------------------------------------- 2000.2: git status (full-v3) 0.19(0.15+0.05) 0.19(0.16+0.05) +0.0% 0.20(0.18+0.03) +5.3% 0.19(0.17+0.04) +0.0% 2000.3: git status (full-v4) 0.20(0.18+0.04) 0.19(0.15+0.06) -5.0% 0.21(0.18+0.05) +5.0% 0.18(0.18+0.02) -10.0% 2000.4: git status (sparse-v3) 0.04(0.04+0.04) 0.05(0.07+0.04) +25.0% 0.04(0.04+0.05) +0.0% 0.04(0.06+0.04) +0.0% 2000.5: git status (sparse-v4) 0.04(0.03+0.06) 0.04(0.05+0.05) +0.0% 0.05(0.05+0.04) +25.0% 0.05(0.06+0.04) +25.0% 2000.6: git add -A (full-v3) 0.36(0.29+0.05) 0.38(0.28+0.07) +5.6% 0.36(0.31+0.05) +0.0% 0.37(0.31+0.05) +2.8% 2000.7: git add -A (full-v4) 0.34(0.27+0.06) 0.34(0.29+0.05) +0.0% 0.34(0.29+0.04) +0.0% 0.35(0.28+0.06) +2.9% 2000.8: git add -A (sparse-v3) 0.06(0.07+0.04) 0.06(0.05+0.06) +0.0% 0.06(0.09+0.01) +0.0% 0.06(0.08+0.03) +0.0% 2000.9: git add -A (sparse-v4) 0.05(0.05+0.04) 0.05(0.05+0.07) +0.0% 0.05(0.04+0.06) +0.0% 0.06(0.06+0.05) +20.0% 2000.10: git add . (full-v3) 0.38(0.31+0.05) 0.37(0.29+0.06) -2.6% 0.37(0.30+0.07) -2.6% 0.37(0.29+0.06) -2.6% 2000.11: git add . (full-v4) 0.35(0.31+0.04) 0.35(0.29+0.07) +0.0% 0.35(0.29+0.05) +0.0% 0.34(0.29+0.06) -2.9% 2000.12: git add . (sparse-v3) 0.06(0.06+0.05) 0.06(0.05+0.06) +0.0% 0.06(0.07+0.05) +0.0% 0.06(0.09+0.03) +0.0% 2000.13: git add . (sparse-v4) 0.06(0.06+0.06) 0.06(0.07+0.04) +0.0% 0.05(0.06+0.05) -16.7% 0.05(0.05+0.07) -16.7% 2000.14: git commit -a -m A (full-v3) 0.48(0.37+0.08) 0.45(0.36+0.08) -6.2% 0.45(0.35+0.09) -6.2% 0.44(0.36+0.07) -8.3% 2000.15: git commit -a -m A (full-v4) 0.45(0.40+0.06) 0.43(0.34+0.07) -4.4% 0.45(0.37+0.06) +0.0% 0.42(0.36+0.05) -6.7% 2000.16: git commit -a -m A (sparse-v3) 0.05(0.05+0.06) 0.05(0.05+0.03) +0.0% 0.05(0.06+0.06) +0.0% 0.05(0.04+0.06) +0.0% 2000.17: git commit -a -m A (sparse-v4) 0.05(0.06+0.03) 0.05(0.06+0.04) +0.0% 0.06(0.07+0.05) +20.0% 0.05(0.04+0.06) +0.0% 2000.18: git checkout -f - (full-v3) 0.55(0.43+0.08) 0.54(0.46+0.05) -1.8% 0.55(0.46+0.07) +0.0% 0.54(0.40+0.10) -1.8% 2000.19: git checkout -f - (full-v4) 0.55(0.41+0.09) 0.50(0.40+0.09) -9.1% 0.51(0.46+0.05) -7.3% 0.51(0.44+0.06) -7.3% 2000.20: git checkout -f - (sparse-v3) 0.06(0.09+0.03) 0.06(0.08+0.03) +0.0% 0.06(0.06+0.05) +0.0% 0.07(0.09+0.03) +16.7% 2000.21: git checkout -f - (sparse-v4) 0.06(0.08+0.04) 0.05(0.07+0.05) -16.7% 0.05(0.07+0.04) -16.7% 0.06(0.09+0.03) +0.0% ``` All of the above were already integrated. ``` 2000.22: git reset (full-v3) 0.41(0.32+0.06) 0.40(0.31+0.06) -2.4% 0.41(0.33+0.05) +0.0% 0.42(0.34+0.04) +2.4% 2000.23: git reset (full-v4) 0.37(0.32+0.05) 0.35(0.30+0.05) -5.4% 0.37(0.30+0.05) +0.0% 0.35(0.31+0.03) -5.4% 2000.24: git reset (sparse-v3) 0.68(0.65+0.05) 0.55(0.52+0.04) -19.1% 0.04(0.05+0.04) -94.1% 0.04(0.05+0.04) -94.1% 2000.25: git reset (sparse-v4) 0.70(0.65+0.05) 0.54(0.50+0.06) -22.9% 0.04(0.07+0.01) -94.3% 0.03(0.05+0.05) -95.7% 2000.26: git reset --hard (full-v3) 0.54(0.43+0.07) 0.53(0.43+0.06) -1.9% 0.55(0.46+0.05) +1.9% 0.55(0.44+0.06) +1.9% 2000.27: git reset --hard (full-v4) 0.50(0.45+0.03) 0.50(0.43+0.05) +0.0% 0.49(0.41+0.06) -2.0% 0.50(0.42+0.05) +0.0% 2000.28: git reset --hard (sparse-v3) 0.83(0.76+0.06) 0.68(0.62+0.05) -18.1% 0.07(0.05+0.02) -91.6% 0.07(0.05+0.02) -91.6% 2000.29: git reset --hard (sparse-v4) 0.80(0.75+0.05) 0.69(0.62+0.06) -13.8% 0.07(0.04+0.02) -91.2% 0.07(0.04+0.03) -91.2% ``` As expected, `git reset [--hard]` improves with the sparse index integration, but remains constant across the full index case. ``` 2000.30: git update-index --add --remove (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.01+0.01) +0.0% 2000.31: git update-index --add --remove (full-v4) 0.03(0.02+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.03+0.00) +0.0% 0.03(0.02+0.01) +0.0% 2000.32: git update-index --add --remove (sparse-v3) 0.57(0.54+0.02) 0.43(0.42+0.00) -24.6% 0.44(0.41+0.03) -22.8% 0.44(0.42+0.01) -22.8% 2000.33: git update-index --add --remove (sparse-v4) 0.56(0.52+0.04) 0.43(0.42+0.01) -23.2% 0.44(0.42+0.02) -21.4% 0.42(0.41+0.01) -25.0% ``` These do not change significantly because #423 is not merged. ``` 2000.34: git diff (full-v3) 0.07(0.05+0.03) 0.06(0.05+0.03) -14.3% 0.07(0.05+0.03) +0.0% 0.06(0.05+0.03) -14.3% 2000.35: git diff (full-v4) 0.06(0.05+0.03) 0.06(0.05+0.02) +0.0% 0.06(0.05+0.02) +0.0% 0.06(0.06+0.02) +0.0% 2000.36: git diff (sparse-v3) 0.25(0.23+0.03) 0.17(0.17+0.02) -32.0% 0.18(0.18+0.02) -28.0% 0.01(0.03+0.03) -96.0% 2000.37: git diff (sparse-v4) 0.25(0.22+0.05) 0.16(0.16+0.01) -36.0% 0.18(0.15+0.04) -28.0% 0.01(0.04+0.02) -96.0% 2000.38: git diff --staged (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.02+0.00) +0.0% 2000.39: git diff --staged (full-v4) 0.04(0.03+0.01) 0.03(0.02+0.01) -25.0% 0.03(0.03+0.00) -25.0% 0.03(0.03+0.00) -25.0% 2000.40: git diff --staged (sparse-v3) 0.21(0.19+0.01) 0.15(0.13+0.01) -28.6% 0.15(0.14+0.01) -28.6% 0.01(0.01+0.00) -95.2% 2000.41: git diff --staged (sparse-v4) 0.22(0.21+0.01) 0.14(0.11+0.03) -36.4% 0.15(0.13+0.02) -31.8% 0.01(0.01+0.00) -95.5% ``` The `git diff` improvements are measurable. ``` 2000.42: git sparse-checkout reapply (full-v3) 0.63(0.54+0.05) 0.56(0.48+0.04) -11.1% 0.57(0.48+0.03) -9.5% 0.59(0.48+0.05) -6.3% 2000.43: git sparse-checkout reapply (full-v4) 0.60(0.54+0.02) 0.51(0.46+0.03) -15.0% 0.54(0.48+0.02) -10.0% 0.50(0.44+0.04) -16.7% 2000.44: git sparse-checkout reapply (sparse-v3) 0.91(0.86+0.05) 0.05(0.05+0.00) -94.5% 0.06(0.05+0.01) -93.4% 0.06(0.06+0.00) -93.4% 2000.45: git sparse-checkout reapply (sparse-v4) 0.92(0.88+0.04) 0.05(0.05+0.00) -94.6% 0.05(0.05+0.01) -94.6% 0.05(0.04+0.01) -94.6% ``` Finally, the `git sparse-checkout` measurements are also present. This test script is particularly valuable when contributing changes upstream. It can be good to start by adding the lines to the performance test in an early commit, then demonstrating the performance change by copying the necessary lines from the output table into your commit message.

Sparse index: integrate with `git update-index`

One thing I forgot when talking about the sparse index is that we have a performance test: `t/perf/p2000-sparse-operations.sh`. This test wasn't helpful for commands like `git merge` that need a particular set of input, but work for more read-only operations. Here is a quick demonstration of how this performance test works so we could have a definitive measure of how your previous updates improved performance. To get these results, I ran the following command in `t/perf`: ``` ./run 4bcd533 f9255a5 f28fc01 b713582 -- p2000-sparse-operations.sh ``` The short-shas correspond to the merge commits for these PRs: * #410 * #421 * #417 * #419 The test takes a copy of the Git repository and creates several copies within a nested directory heirarchy. ``` Test 4bcd533 f9255a5 f28fc01 b713582 ------------------------------------------------------------------------------------------------------------------------------------------------- 2000.2: git status (full-v3) 0.19(0.15+0.05) 0.19(0.16+0.05) +0.0% 0.20(0.18+0.03) +5.3% 0.19(0.17+0.04) +0.0% 2000.3: git status (full-v4) 0.20(0.18+0.04) 0.19(0.15+0.06) -5.0% 0.21(0.18+0.05) +5.0% 0.18(0.18+0.02) -10.0% 2000.4: git status (sparse-v3) 0.04(0.04+0.04) 0.05(0.07+0.04) +25.0% 0.04(0.04+0.05) +0.0% 0.04(0.06+0.04) +0.0% 2000.5: git status (sparse-v4) 0.04(0.03+0.06) 0.04(0.05+0.05) +0.0% 0.05(0.05+0.04) +25.0% 0.05(0.06+0.04) +25.0% 2000.6: git add -A (full-v3) 0.36(0.29+0.05) 0.38(0.28+0.07) +5.6% 0.36(0.31+0.05) +0.0% 0.37(0.31+0.05) +2.8% 2000.7: git add -A (full-v4) 0.34(0.27+0.06) 0.34(0.29+0.05) +0.0% 0.34(0.29+0.04) +0.0% 0.35(0.28+0.06) +2.9% 2000.8: git add -A (sparse-v3) 0.06(0.07+0.04) 0.06(0.05+0.06) +0.0% 0.06(0.09+0.01) +0.0% 0.06(0.08+0.03) +0.0% 2000.9: git add -A (sparse-v4) 0.05(0.05+0.04) 0.05(0.05+0.07) +0.0% 0.05(0.04+0.06) +0.0% 0.06(0.06+0.05) +20.0% 2000.10: git add . (full-v3) 0.38(0.31+0.05) 0.37(0.29+0.06) -2.6% 0.37(0.30+0.07) -2.6% 0.37(0.29+0.06) -2.6% 2000.11: git add . (full-v4) 0.35(0.31+0.04) 0.35(0.29+0.07) +0.0% 0.35(0.29+0.05) +0.0% 0.34(0.29+0.06) -2.9% 2000.12: git add . (sparse-v3) 0.06(0.06+0.05) 0.06(0.05+0.06) +0.0% 0.06(0.07+0.05) +0.0% 0.06(0.09+0.03) +0.0% 2000.13: git add . (sparse-v4) 0.06(0.06+0.06) 0.06(0.07+0.04) +0.0% 0.05(0.06+0.05) -16.7% 0.05(0.05+0.07) -16.7% 2000.14: git commit -a -m A (full-v3) 0.48(0.37+0.08) 0.45(0.36+0.08) -6.2% 0.45(0.35+0.09) -6.2% 0.44(0.36+0.07) -8.3% 2000.15: git commit -a -m A (full-v4) 0.45(0.40+0.06) 0.43(0.34+0.07) -4.4% 0.45(0.37+0.06) +0.0% 0.42(0.36+0.05) -6.7% 2000.16: git commit -a -m A (sparse-v3) 0.05(0.05+0.06) 0.05(0.05+0.03) +0.0% 0.05(0.06+0.06) +0.0% 0.05(0.04+0.06) +0.0% 2000.17: git commit -a -m A (sparse-v4) 0.05(0.06+0.03) 0.05(0.06+0.04) +0.0% 0.06(0.07+0.05) +20.0% 0.05(0.04+0.06) +0.0% 2000.18: git checkout -f - (full-v3) 0.55(0.43+0.08) 0.54(0.46+0.05) -1.8% 0.55(0.46+0.07) +0.0% 0.54(0.40+0.10) -1.8% 2000.19: git checkout -f - (full-v4) 0.55(0.41+0.09) 0.50(0.40+0.09) -9.1% 0.51(0.46+0.05) -7.3% 0.51(0.44+0.06) -7.3% 2000.20: git checkout -f - (sparse-v3) 0.06(0.09+0.03) 0.06(0.08+0.03) +0.0% 0.06(0.06+0.05) +0.0% 0.07(0.09+0.03) +16.7% 2000.21: git checkout -f - (sparse-v4) 0.06(0.08+0.04) 0.05(0.07+0.05) -16.7% 0.05(0.07+0.04) -16.7% 0.06(0.09+0.03) +0.0% ``` All of the above were already integrated. ``` 2000.22: git reset (full-v3) 0.41(0.32+0.06) 0.40(0.31+0.06) -2.4% 0.41(0.33+0.05) +0.0% 0.42(0.34+0.04) +2.4% 2000.23: git reset (full-v4) 0.37(0.32+0.05) 0.35(0.30+0.05) -5.4% 0.37(0.30+0.05) +0.0% 0.35(0.31+0.03) -5.4% 2000.24: git reset (sparse-v3) 0.68(0.65+0.05) 0.55(0.52+0.04) -19.1% 0.04(0.05+0.04) -94.1% 0.04(0.05+0.04) -94.1% 2000.25: git reset (sparse-v4) 0.70(0.65+0.05) 0.54(0.50+0.06) -22.9% 0.04(0.07+0.01) -94.3% 0.03(0.05+0.05) -95.7% 2000.26: git reset --hard (full-v3) 0.54(0.43+0.07) 0.53(0.43+0.06) -1.9% 0.55(0.46+0.05) +1.9% 0.55(0.44+0.06) +1.9% 2000.27: git reset --hard (full-v4) 0.50(0.45+0.03) 0.50(0.43+0.05) +0.0% 0.49(0.41+0.06) -2.0% 0.50(0.42+0.05) +0.0% 2000.28: git reset --hard (sparse-v3) 0.83(0.76+0.06) 0.68(0.62+0.05) -18.1% 0.07(0.05+0.02) -91.6% 0.07(0.05+0.02) -91.6% 2000.29: git reset --hard (sparse-v4) 0.80(0.75+0.05) 0.69(0.62+0.06) -13.8% 0.07(0.04+0.02) -91.2% 0.07(0.04+0.03) -91.2% ``` As expected, `git reset [--hard]` improves with the sparse index integration, but remains constant across the full index case. ``` 2000.30: git update-index --add --remove (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.01+0.01) +0.0% 2000.31: git update-index --add --remove (full-v4) 0.03(0.02+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.03+0.00) +0.0% 0.03(0.02+0.01) +0.0% 2000.32: git update-index --add --remove (sparse-v3) 0.57(0.54+0.02) 0.43(0.42+0.00) -24.6% 0.44(0.41+0.03) -22.8% 0.44(0.42+0.01) -22.8% 2000.33: git update-index --add --remove (sparse-v4) 0.56(0.52+0.04) 0.43(0.42+0.01) -23.2% 0.44(0.42+0.02) -21.4% 0.42(0.41+0.01) -25.0% ``` These do not change significantly because #423 is not merged. ``` 2000.34: git diff (full-v3) 0.07(0.05+0.03) 0.06(0.05+0.03) -14.3% 0.07(0.05+0.03) +0.0% 0.06(0.05+0.03) -14.3% 2000.35: git diff (full-v4) 0.06(0.05+0.03) 0.06(0.05+0.02) +0.0% 0.06(0.05+0.02) +0.0% 0.06(0.06+0.02) +0.0% 2000.36: git diff (sparse-v3) 0.25(0.23+0.03) 0.17(0.17+0.02) -32.0% 0.18(0.18+0.02) -28.0% 0.01(0.03+0.03) -96.0% 2000.37: git diff (sparse-v4) 0.25(0.22+0.05) 0.16(0.16+0.01) -36.0% 0.18(0.15+0.04) -28.0% 0.01(0.04+0.02) -96.0% 2000.38: git diff --staged (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.02+0.00) +0.0% 2000.39: git diff --staged (full-v4) 0.04(0.03+0.01) 0.03(0.02+0.01) -25.0% 0.03(0.03+0.00) -25.0% 0.03(0.03+0.00) -25.0% 2000.40: git diff --staged (sparse-v3) 0.21(0.19+0.01) 0.15(0.13+0.01) -28.6% 0.15(0.14+0.01) -28.6% 0.01(0.01+0.00) -95.2% 2000.41: git diff --staged (sparse-v4) 0.22(0.21+0.01) 0.14(0.11+0.03) -36.4% 0.15(0.13+0.02) -31.8% 0.01(0.01+0.00) -95.5% ``` The `git diff` improvements are measurable. ``` 2000.42: git sparse-checkout reapply (full-v3) 0.63(0.54+0.05) 0.56(0.48+0.04) -11.1% 0.57(0.48+0.03) -9.5% 0.59(0.48+0.05) -6.3% 2000.43: git sparse-checkout reapply (full-v4) 0.60(0.54+0.02) 0.51(0.46+0.03) -15.0% 0.54(0.48+0.02) -10.0% 0.50(0.44+0.04) -16.7% 2000.44: git sparse-checkout reapply (sparse-v3) 0.91(0.86+0.05) 0.05(0.05+0.00) -94.5% 0.06(0.05+0.01) -93.4% 0.06(0.06+0.00) -93.4% 2000.45: git sparse-checkout reapply (sparse-v4) 0.92(0.88+0.04) 0.05(0.05+0.00) -94.6% 0.05(0.05+0.01) -94.6% 0.05(0.04+0.01) -94.6% ``` Finally, the `git sparse-checkout` measurements are also present. This test script is particularly valuable when contributing changes upstream. It can be good to start by adding the lines to the performance test in an early commit, then demonstrating the performance change by copying the necessary lines from the output table into your commit message.

Sparse index: integrate with `git update-index`

One thing I forgot when talking about the sparse index is that we have a performance test: `t/perf/p2000-sparse-operations.sh`. This test wasn't helpful for commands like `git merge` that need a particular set of input, but work for more read-only operations. Here is a quick demonstration of how this performance test works so we could have a definitive measure of how your previous updates improved performance. To get these results, I ran the following command in `t/perf`: ``` ./run 4bcd533 f9255a5 f28fc01 b713582 -- p2000-sparse-operations.sh ``` The short-shas correspond to the merge commits for these PRs: * #410 * #421 * #417 * #419 The test takes a copy of the Git repository and creates several copies within a nested directory heirarchy. ``` Test 4bcd533 f9255a5 f28fc01 b713582 ------------------------------------------------------------------------------------------------------------------------------------------------- 2000.2: git status (full-v3) 0.19(0.15+0.05) 0.19(0.16+0.05) +0.0% 0.20(0.18+0.03) +5.3% 0.19(0.17+0.04) +0.0% 2000.3: git status (full-v4) 0.20(0.18+0.04) 0.19(0.15+0.06) -5.0% 0.21(0.18+0.05) +5.0% 0.18(0.18+0.02) -10.0% 2000.4: git status (sparse-v3) 0.04(0.04+0.04) 0.05(0.07+0.04) +25.0% 0.04(0.04+0.05) +0.0% 0.04(0.06+0.04) +0.0% 2000.5: git status (sparse-v4) 0.04(0.03+0.06) 0.04(0.05+0.05) +0.0% 0.05(0.05+0.04) +25.0% 0.05(0.06+0.04) +25.0% 2000.6: git add -A (full-v3) 0.36(0.29+0.05) 0.38(0.28+0.07) +5.6% 0.36(0.31+0.05) +0.0% 0.37(0.31+0.05) +2.8% 2000.7: git add -A (full-v4) 0.34(0.27+0.06) 0.34(0.29+0.05) +0.0% 0.34(0.29+0.04) +0.0% 0.35(0.28+0.06) +2.9% 2000.8: git add -A (sparse-v3) 0.06(0.07+0.04) 0.06(0.05+0.06) +0.0% 0.06(0.09+0.01) +0.0% 0.06(0.08+0.03) +0.0% 2000.9: git add -A (sparse-v4) 0.05(0.05+0.04) 0.05(0.05+0.07) +0.0% 0.05(0.04+0.06) +0.0% 0.06(0.06+0.05) +20.0% 2000.10: git add . (full-v3) 0.38(0.31+0.05) 0.37(0.29+0.06) -2.6% 0.37(0.30+0.07) -2.6% 0.37(0.29+0.06) -2.6% 2000.11: git add . (full-v4) 0.35(0.31+0.04) 0.35(0.29+0.07) +0.0% 0.35(0.29+0.05) +0.0% 0.34(0.29+0.06) -2.9% 2000.12: git add . (sparse-v3) 0.06(0.06+0.05) 0.06(0.05+0.06) +0.0% 0.06(0.07+0.05) +0.0% 0.06(0.09+0.03) +0.0% 2000.13: git add . (sparse-v4) 0.06(0.06+0.06) 0.06(0.07+0.04) +0.0% 0.05(0.06+0.05) -16.7% 0.05(0.05+0.07) -16.7% 2000.14: git commit -a -m A (full-v3) 0.48(0.37+0.08) 0.45(0.36+0.08) -6.2% 0.45(0.35+0.09) -6.2% 0.44(0.36+0.07) -8.3% 2000.15: git commit -a -m A (full-v4) 0.45(0.40+0.06) 0.43(0.34+0.07) -4.4% 0.45(0.37+0.06) +0.0% 0.42(0.36+0.05) -6.7% 2000.16: git commit -a -m A (sparse-v3) 0.05(0.05+0.06) 0.05(0.05+0.03) +0.0% 0.05(0.06+0.06) +0.0% 0.05(0.04+0.06) +0.0% 2000.17: git commit -a -m A (sparse-v4) 0.05(0.06+0.03) 0.05(0.06+0.04) +0.0% 0.06(0.07+0.05) +20.0% 0.05(0.04+0.06) +0.0% 2000.18: git checkout -f - (full-v3) 0.55(0.43+0.08) 0.54(0.46+0.05) -1.8% 0.55(0.46+0.07) +0.0% 0.54(0.40+0.10) -1.8% 2000.19: git checkout -f - (full-v4) 0.55(0.41+0.09) 0.50(0.40+0.09) -9.1% 0.51(0.46+0.05) -7.3% 0.51(0.44+0.06) -7.3% 2000.20: git checkout -f - (sparse-v3) 0.06(0.09+0.03) 0.06(0.08+0.03) +0.0% 0.06(0.06+0.05) +0.0% 0.07(0.09+0.03) +16.7% 2000.21: git checkout -f - (sparse-v4) 0.06(0.08+0.04) 0.05(0.07+0.05) -16.7% 0.05(0.07+0.04) -16.7% 0.06(0.09+0.03) +0.0% ``` All of the above were already integrated. ``` 2000.22: git reset (full-v3) 0.41(0.32+0.06) 0.40(0.31+0.06) -2.4% 0.41(0.33+0.05) +0.0% 0.42(0.34+0.04) +2.4% 2000.23: git reset (full-v4) 0.37(0.32+0.05) 0.35(0.30+0.05) -5.4% 0.37(0.30+0.05) +0.0% 0.35(0.31+0.03) -5.4% 2000.24: git reset (sparse-v3) 0.68(0.65+0.05) 0.55(0.52+0.04) -19.1% 0.04(0.05+0.04) -94.1% 0.04(0.05+0.04) -94.1% 2000.25: git reset (sparse-v4) 0.70(0.65+0.05) 0.54(0.50+0.06) -22.9% 0.04(0.07+0.01) -94.3% 0.03(0.05+0.05) -95.7% 2000.26: git reset --hard (full-v3) 0.54(0.43+0.07) 0.53(0.43+0.06) -1.9% 0.55(0.46+0.05) +1.9% 0.55(0.44+0.06) +1.9% 2000.27: git reset --hard (full-v4) 0.50(0.45+0.03) 0.50(0.43+0.05) +0.0% 0.49(0.41+0.06) -2.0% 0.50(0.42+0.05) +0.0% 2000.28: git reset --hard (sparse-v3) 0.83(0.76+0.06) 0.68(0.62+0.05) -18.1% 0.07(0.05+0.02) -91.6% 0.07(0.05+0.02) -91.6% 2000.29: git reset --hard (sparse-v4) 0.80(0.75+0.05) 0.69(0.62+0.06) -13.8% 0.07(0.04+0.02) -91.2% 0.07(0.04+0.03) -91.2% ``` As expected, `git reset [--hard]` improves with the sparse index integration, but remains constant across the full index case. ``` 2000.30: git update-index --add --remove (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.01+0.01) +0.0% 2000.31: git update-index --add --remove (full-v4) 0.03(0.02+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.03+0.00) +0.0% 0.03(0.02+0.01) +0.0% 2000.32: git update-index --add --remove (sparse-v3) 0.57(0.54+0.02) 0.43(0.42+0.00) -24.6% 0.44(0.41+0.03) -22.8% 0.44(0.42+0.01) -22.8% 2000.33: git update-index --add --remove (sparse-v4) 0.56(0.52+0.04) 0.43(0.42+0.01) -23.2% 0.44(0.42+0.02) -21.4% 0.42(0.41+0.01) -25.0% ``` These do not change significantly because #423 is not merged. ``` 2000.34: git diff (full-v3) 0.07(0.05+0.03) 0.06(0.05+0.03) -14.3% 0.07(0.05+0.03) +0.0% 0.06(0.05+0.03) -14.3% 2000.35: git diff (full-v4) 0.06(0.05+0.03) 0.06(0.05+0.02) +0.0% 0.06(0.05+0.02) +0.0% 0.06(0.06+0.02) +0.0% 2000.36: git diff (sparse-v3) 0.25(0.23+0.03) 0.17(0.17+0.02) -32.0% 0.18(0.18+0.02) -28.0% 0.01(0.03+0.03) -96.0% 2000.37: git diff (sparse-v4) 0.25(0.22+0.05) 0.16(0.16+0.01) -36.0% 0.18(0.15+0.04) -28.0% 0.01(0.04+0.02) -96.0% 2000.38: git diff --staged (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.02+0.00) +0.0% 2000.39: git diff --staged (full-v4) 0.04(0.03+0.01) 0.03(0.02+0.01) -25.0% 0.03(0.03+0.00) -25.0% 0.03(0.03+0.00) -25.0% 2000.40: git diff --staged (sparse-v3) 0.21(0.19+0.01) 0.15(0.13+0.01) -28.6% 0.15(0.14+0.01) -28.6% 0.01(0.01+0.00) -95.2% 2000.41: git diff --staged (sparse-v4) 0.22(0.21+0.01) 0.14(0.11+0.03) -36.4% 0.15(0.13+0.02) -31.8% 0.01(0.01+0.00) -95.5% ``` The `git diff` improvements are measurable. ``` 2000.42: git sparse-checkout reapply (full-v3) 0.63(0.54+0.05) 0.56(0.48+0.04) -11.1% 0.57(0.48+0.03) -9.5% 0.59(0.48+0.05) -6.3% 2000.43: git sparse-checkout reapply (full-v4) 0.60(0.54+0.02) 0.51(0.46+0.03) -15.0% 0.54(0.48+0.02) -10.0% 0.50(0.44+0.04) -16.7% 2000.44: git sparse-checkout reapply (sparse-v3) 0.91(0.86+0.05) 0.05(0.05+0.00) -94.5% 0.06(0.05+0.01) -93.4% 0.06(0.06+0.00) -93.4% 2000.45: git sparse-checkout reapply (sparse-v4) 0.92(0.88+0.04) 0.05(0.05+0.00) -94.6% 0.05(0.05+0.01) -94.6% 0.05(0.04+0.01) -94.6% ``` Finally, the `git sparse-checkout` measurements are also present. This test script is particularly valuable when contributing changes upstream. It can be good to start by adding the lines to the performance test in an early commit, then demonstrating the performance change by copying the necessary lines from the output table into your commit message.

Sparse index: integrate with `git update-index`

One thing I forgot when talking about the sparse index is that we have a performance test: `t/perf/p2000-sparse-operations.sh`. This test wasn't helpful for commands like `git merge` that need a particular set of input, but work for more read-only operations. Here is a quick demonstration of how this performance test works so we could have a definitive measure of how your previous updates improved performance. To get these results, I ran the following command in `t/perf`: ``` ./run 4bcd533 f9255a5 f28fc01 b713582 -- p2000-sparse-operations.sh ``` The short-shas correspond to the merge commits for these PRs: * #410 * #421 * #417 * #419 The test takes a copy of the Git repository and creates several copies within a nested directory heirarchy. ``` Test 4bcd533 f9255a5 f28fc01 b713582 ------------------------------------------------------------------------------------------------------------------------------------------------- 2000.2: git status (full-v3) 0.19(0.15+0.05) 0.19(0.16+0.05) +0.0% 0.20(0.18+0.03) +5.3% 0.19(0.17+0.04) +0.0% 2000.3: git status (full-v4) 0.20(0.18+0.04) 0.19(0.15+0.06) -5.0% 0.21(0.18+0.05) +5.0% 0.18(0.18+0.02) -10.0% 2000.4: git status (sparse-v3) 0.04(0.04+0.04) 0.05(0.07+0.04) +25.0% 0.04(0.04+0.05) +0.0% 0.04(0.06+0.04) +0.0% 2000.5: git status (sparse-v4) 0.04(0.03+0.06) 0.04(0.05+0.05) +0.0% 0.05(0.05+0.04) +25.0% 0.05(0.06+0.04) +25.0% 2000.6: git add -A (full-v3) 0.36(0.29+0.05) 0.38(0.28+0.07) +5.6% 0.36(0.31+0.05) +0.0% 0.37(0.31+0.05) +2.8% 2000.7: git add -A (full-v4) 0.34(0.27+0.06) 0.34(0.29+0.05) +0.0% 0.34(0.29+0.04) +0.0% 0.35(0.28+0.06) +2.9% 2000.8: git add -A (sparse-v3) 0.06(0.07+0.04) 0.06(0.05+0.06) +0.0% 0.06(0.09+0.01) +0.0% 0.06(0.08+0.03) +0.0% 2000.9: git add -A (sparse-v4) 0.05(0.05+0.04) 0.05(0.05+0.07) +0.0% 0.05(0.04+0.06) +0.0% 0.06(0.06+0.05) +20.0% 2000.10: git add . (full-v3) 0.38(0.31+0.05) 0.37(0.29+0.06) -2.6% 0.37(0.30+0.07) -2.6% 0.37(0.29+0.06) -2.6% 2000.11: git add . (full-v4) 0.35(0.31+0.04) 0.35(0.29+0.07) +0.0% 0.35(0.29+0.05) +0.0% 0.34(0.29+0.06) -2.9% 2000.12: git add . (sparse-v3) 0.06(0.06+0.05) 0.06(0.05+0.06) +0.0% 0.06(0.07+0.05) +0.0% 0.06(0.09+0.03) +0.0% 2000.13: git add . (sparse-v4) 0.06(0.06+0.06) 0.06(0.07+0.04) +0.0% 0.05(0.06+0.05) -16.7% 0.05(0.05+0.07) -16.7% 2000.14: git commit -a -m A (full-v3) 0.48(0.37+0.08) 0.45(0.36+0.08) -6.2% 0.45(0.35+0.09) -6.2% 0.44(0.36+0.07) -8.3% 2000.15: git commit -a -m A (full-v4) 0.45(0.40+0.06) 0.43(0.34+0.07) -4.4% 0.45(0.37+0.06) +0.0% 0.42(0.36+0.05) -6.7% 2000.16: git commit -a -m A (sparse-v3) 0.05(0.05+0.06) 0.05(0.05+0.03) +0.0% 0.05(0.06+0.06) +0.0% 0.05(0.04+0.06) +0.0% 2000.17: git commit -a -m A (sparse-v4) 0.05(0.06+0.03) 0.05(0.06+0.04) +0.0% 0.06(0.07+0.05) +20.0% 0.05(0.04+0.06) +0.0% 2000.18: git checkout -f - (full-v3) 0.55(0.43+0.08) 0.54(0.46+0.05) -1.8% 0.55(0.46+0.07) +0.0% 0.54(0.40+0.10) -1.8% 2000.19: git checkout -f - (full-v4) 0.55(0.41+0.09) 0.50(0.40+0.09) -9.1% 0.51(0.46+0.05) -7.3% 0.51(0.44+0.06) -7.3% 2000.20: git checkout -f - (sparse-v3) 0.06(0.09+0.03) 0.06(0.08+0.03) +0.0% 0.06(0.06+0.05) +0.0% 0.07(0.09+0.03) +16.7% 2000.21: git checkout -f - (sparse-v4) 0.06(0.08+0.04) 0.05(0.07+0.05) -16.7% 0.05(0.07+0.04) -16.7% 0.06(0.09+0.03) +0.0% ``` All of the above were already integrated. ``` 2000.22: git reset (full-v3) 0.41(0.32+0.06) 0.40(0.31+0.06) -2.4% 0.41(0.33+0.05) +0.0% 0.42(0.34+0.04) +2.4% 2000.23: git reset (full-v4) 0.37(0.32+0.05) 0.35(0.30+0.05) -5.4% 0.37(0.30+0.05) +0.0% 0.35(0.31+0.03) -5.4% 2000.24: git reset (sparse-v3) 0.68(0.65+0.05) 0.55(0.52+0.04) -19.1% 0.04(0.05+0.04) -94.1% 0.04(0.05+0.04) -94.1% 2000.25: git reset (sparse-v4) 0.70(0.65+0.05) 0.54(0.50+0.06) -22.9% 0.04(0.07+0.01) -94.3% 0.03(0.05+0.05) -95.7% 2000.26: git reset --hard (full-v3) 0.54(0.43+0.07) 0.53(0.43+0.06) -1.9% 0.55(0.46+0.05) +1.9% 0.55(0.44+0.06) +1.9% 2000.27: git reset --hard (full-v4) 0.50(0.45+0.03) 0.50(0.43+0.05) +0.0% 0.49(0.41+0.06) -2.0% 0.50(0.42+0.05) +0.0% 2000.28: git reset --hard (sparse-v3) 0.83(0.76+0.06) 0.68(0.62+0.05) -18.1% 0.07(0.05+0.02) -91.6% 0.07(0.05+0.02) -91.6% 2000.29: git reset --hard (sparse-v4) 0.80(0.75+0.05) 0.69(0.62+0.06) -13.8% 0.07(0.04+0.02) -91.2% 0.07(0.04+0.03) -91.2% ``` As expected, `git reset [--hard]` improves with the sparse index integration, but remains constant across the full index case. ``` 2000.30: git update-index --add --remove (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.01+0.01) +0.0% 2000.31: git update-index --add --remove (full-v4) 0.03(0.02+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.03+0.00) +0.0% 0.03(0.02+0.01) +0.0% 2000.32: git update-index --add --remove (sparse-v3) 0.57(0.54+0.02) 0.43(0.42+0.00) -24.6% 0.44(0.41+0.03) -22.8% 0.44(0.42+0.01) -22.8% 2000.33: git update-index --add --remove (sparse-v4) 0.56(0.52+0.04) 0.43(0.42+0.01) -23.2% 0.44(0.42+0.02) -21.4% 0.42(0.41+0.01) -25.0% ``` These do not change significantly because #423 is not merged. ``` 2000.34: git diff (full-v3) 0.07(0.05+0.03) 0.06(0.05+0.03) -14.3% 0.07(0.05+0.03) +0.0% 0.06(0.05+0.03) -14.3% 2000.35: git diff (full-v4) 0.06(0.05+0.03) 0.06(0.05+0.02) +0.0% 0.06(0.05+0.02) +0.0% 0.06(0.06+0.02) +0.0% 2000.36: git diff (sparse-v3) 0.25(0.23+0.03) 0.17(0.17+0.02) -32.0% 0.18(0.18+0.02) -28.0% 0.01(0.03+0.03) -96.0% 2000.37: git diff (sparse-v4) 0.25(0.22+0.05) 0.16(0.16+0.01) -36.0% 0.18(0.15+0.04) -28.0% 0.01(0.04+0.02) -96.0% 2000.38: git diff --staged (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.02+0.00) +0.0% 2000.39: git diff --staged (full-v4) 0.04(0.03+0.01) 0.03(0.02+0.01) -25.0% 0.03(0.03+0.00) -25.0% 0.03(0.03+0.00) -25.0% 2000.40: git diff --staged (sparse-v3) 0.21(0.19+0.01) 0.15(0.13+0.01) -28.6% 0.15(0.14+0.01) -28.6% 0.01(0.01+0.00) -95.2% 2000.41: git diff --staged (sparse-v4) 0.22(0.21+0.01) 0.14(0.11+0.03) -36.4% 0.15(0.13+0.02) -31.8% 0.01(0.01+0.00) -95.5% ``` The `git diff` improvements are measurable. ``` 2000.42: git sparse-checkout reapply (full-v3) 0.63(0.54+0.05) 0.56(0.48+0.04) -11.1% 0.57(0.48+0.03) -9.5% 0.59(0.48+0.05) -6.3% 2000.43: git sparse-checkout reapply (full-v4) 0.60(0.54+0.02) 0.51(0.46+0.03) -15.0% 0.54(0.48+0.02) -10.0% 0.50(0.44+0.04) -16.7% 2000.44: git sparse-checkout reapply (sparse-v3) 0.91(0.86+0.05) 0.05(0.05+0.00) -94.5% 0.06(0.05+0.01) -93.4% 0.06(0.06+0.00) -93.4% 2000.45: git sparse-checkout reapply (sparse-v4) 0.92(0.88+0.04) 0.05(0.05+0.00) -94.6% 0.05(0.05+0.01) -94.6% 0.05(0.04+0.01) -94.6% ``` Finally, the `git sparse-checkout` measurements are also present. This test script is particularly valuable when contributing changes upstream. It can be good to start by adding the lines to the performance test in an early commit, then demonstrating the performance change by copying the necessary lines from the output table into your commit message.

…tests One thing I forgot when talking about the sparse index is that we have a performance test: `t/perf/p2000-sparse-operations.sh`. This test wasn't helpful for commands like `git merge` that need a particular set of input, but work for more read-only operations. Here is a quick demonstration of how this performance test works so we could have a definitive measure of how your previous updates improved performance. To get these results, I ran the following command in `t/perf`: ``` ./run 4bcd533 f9255a5 f28fc01 b713582 -- p2000-sparse-operations.sh ``` The short-shas correspond to the merge commits for these PRs: * microsoft#410 * microsoft#421 * microsoft#417 * microsoft#419 The test takes a copy of the Git repository and creates several copies within a nested directory heirarchy. ``` Test 4bcd533 f9255a5 f28fc01 b713582 ------------------------------------------------------------------------------------------------------------------------------------------------- 2000.2: git status (full-v3) 0.19(0.15+0.05) 0.19(0.16+0.05) +0.0% 0.20(0.18+0.03) +5.3% 0.19(0.17+0.04) +0.0% 2000.3: git status (full-v4) 0.20(0.18+0.04) 0.19(0.15+0.06) -5.0% 0.21(0.18+0.05) +5.0% 0.18(0.18+0.02) -10.0% 2000.4: git status (sparse-v3) 0.04(0.04+0.04) 0.05(0.07+0.04) +25.0% 0.04(0.04+0.05) +0.0% 0.04(0.06+0.04) +0.0% 2000.5: git status (sparse-v4) 0.04(0.03+0.06) 0.04(0.05+0.05) +0.0% 0.05(0.05+0.04) +25.0% 0.05(0.06+0.04) +25.0% 2000.6: git add -A (full-v3) 0.36(0.29+0.05) 0.38(0.28+0.07) +5.6% 0.36(0.31+0.05) +0.0% 0.37(0.31+0.05) +2.8% 2000.7: git add -A (full-v4) 0.34(0.27+0.06) 0.34(0.29+0.05) +0.0% 0.34(0.29+0.04) +0.0% 0.35(0.28+0.06) +2.9% 2000.8: git add -A (sparse-v3) 0.06(0.07+0.04) 0.06(0.05+0.06) +0.0% 0.06(0.09+0.01) +0.0% 0.06(0.08+0.03) +0.0% 2000.9: git add -A (sparse-v4) 0.05(0.05+0.04) 0.05(0.05+0.07) +0.0% 0.05(0.04+0.06) +0.0% 0.06(0.06+0.05) +20.0% 2000.10: git add . (full-v3) 0.38(0.31+0.05) 0.37(0.29+0.06) -2.6% 0.37(0.30+0.07) -2.6% 0.37(0.29+0.06) -2.6% 2000.11: git add . (full-v4) 0.35(0.31+0.04) 0.35(0.29+0.07) +0.0% 0.35(0.29+0.05) +0.0% 0.34(0.29+0.06) -2.9% 2000.12: git add . (sparse-v3) 0.06(0.06+0.05) 0.06(0.05+0.06) +0.0% 0.06(0.07+0.05) +0.0% 0.06(0.09+0.03) +0.0% 2000.13: git add . (sparse-v4) 0.06(0.06+0.06) 0.06(0.07+0.04) +0.0% 0.05(0.06+0.05) -16.7% 0.05(0.05+0.07) -16.7% 2000.14: git commit -a -m A (full-v3) 0.48(0.37+0.08) 0.45(0.36+0.08) -6.2% 0.45(0.35+0.09) -6.2% 0.44(0.36+0.07) -8.3% 2000.15: git commit -a -m A (full-v4) 0.45(0.40+0.06) 0.43(0.34+0.07) -4.4% 0.45(0.37+0.06) +0.0% 0.42(0.36+0.05) -6.7% 2000.16: git commit -a -m A (sparse-v3) 0.05(0.05+0.06) 0.05(0.05+0.03) +0.0% 0.05(0.06+0.06) +0.0% 0.05(0.04+0.06) +0.0% 2000.17: git commit -a -m A (sparse-v4) 0.05(0.06+0.03) 0.05(0.06+0.04) +0.0% 0.06(0.07+0.05) +20.0% 0.05(0.04+0.06) +0.0% 2000.18: git checkout -f - (full-v3) 0.55(0.43+0.08) 0.54(0.46+0.05) -1.8% 0.55(0.46+0.07) +0.0% 0.54(0.40+0.10) -1.8% 2000.19: git checkout -f - (full-v4) 0.55(0.41+0.09) 0.50(0.40+0.09) -9.1% 0.51(0.46+0.05) -7.3% 0.51(0.44+0.06) -7.3% 2000.20: git checkout -f - (sparse-v3) 0.06(0.09+0.03) 0.06(0.08+0.03) +0.0% 0.06(0.06+0.05) +0.0% 0.07(0.09+0.03) +16.7% 2000.21: git checkout -f - (sparse-v4) 0.06(0.08+0.04) 0.05(0.07+0.05) -16.7% 0.05(0.07+0.04) -16.7% 0.06(0.09+0.03) +0.0% ``` All of the above were already integrated. ``` 2000.22: git reset (full-v3) 0.41(0.32+0.06) 0.40(0.31+0.06) -2.4% 0.41(0.33+0.05) +0.0% 0.42(0.34+0.04) +2.4% 2000.23: git reset (full-v4) 0.37(0.32+0.05) 0.35(0.30+0.05) -5.4% 0.37(0.30+0.05) +0.0% 0.35(0.31+0.03) -5.4% 2000.24: git reset (sparse-v3) 0.68(0.65+0.05) 0.55(0.52+0.04) -19.1% 0.04(0.05+0.04) -94.1% 0.04(0.05+0.04) -94.1% 2000.25: git reset (sparse-v4) 0.70(0.65+0.05) 0.54(0.50+0.06) -22.9% 0.04(0.07+0.01) -94.3% 0.03(0.05+0.05) -95.7% 2000.26: git reset --hard (full-v3) 0.54(0.43+0.07) 0.53(0.43+0.06) -1.9% 0.55(0.46+0.05) +1.9% 0.55(0.44+0.06) +1.9% 2000.27: git reset --hard (full-v4) 0.50(0.45+0.03) 0.50(0.43+0.05) +0.0% 0.49(0.41+0.06) -2.0% 0.50(0.42+0.05) +0.0% 2000.28: git reset --hard (sparse-v3) 0.83(0.76+0.06) 0.68(0.62+0.05) -18.1% 0.07(0.05+0.02) -91.6% 0.07(0.05+0.02) -91.6% 2000.29: git reset --hard (sparse-v4) 0.80(0.75+0.05) 0.69(0.62+0.06) -13.8% 0.07(0.04+0.02) -91.2% 0.07(0.04+0.03) -91.2% ``` As expected, `git reset [--hard]` improves with the sparse index integration, but remains constant across the full index case. ``` 2000.30: git update-index --add --remove (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.01+0.01) +0.0% 2000.31: git update-index --add --remove (full-v4) 0.03(0.02+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.03+0.00) +0.0% 0.03(0.02+0.01) +0.0% 2000.32: git update-index --add --remove (sparse-v3) 0.57(0.54+0.02) 0.43(0.42+0.00) -24.6% 0.44(0.41+0.03) -22.8% 0.44(0.42+0.01) -22.8% 2000.33: git update-index --add --remove (sparse-v4) 0.56(0.52+0.04) 0.43(0.42+0.01) -23.2% 0.44(0.42+0.02) -21.4% 0.42(0.41+0.01) -25.0% ``` These do not change significantly because microsoft#423 is not merged. ``` 2000.34: git diff (full-v3) 0.07(0.05+0.03) 0.06(0.05+0.03) -14.3% 0.07(0.05+0.03) +0.0% 0.06(0.05+0.03) -14.3% 2000.35: git diff (full-v4) 0.06(0.05+0.03) 0.06(0.05+0.02) +0.0% 0.06(0.05+0.02) +0.0% 0.06(0.06+0.02) +0.0% 2000.36: git diff (sparse-v3) 0.25(0.23+0.03) 0.17(0.17+0.02) -32.0% 0.18(0.18+0.02) -28.0% 0.01(0.03+0.03) -96.0% 2000.37: git diff (sparse-v4) 0.25(0.22+0.05) 0.16(0.16+0.01) -36.0% 0.18(0.15+0.04) -28.0% 0.01(0.04+0.02) -96.0% 2000.38: git diff --staged (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.02+0.00) +0.0% 2000.39: git diff --staged (full-v4) 0.04(0.03+0.01) 0.03(0.02+0.01) -25.0% 0.03(0.03+0.00) -25.0% 0.03(0.03+0.00) -25.0% 2000.40: git diff --staged (sparse-v3) 0.21(0.19+0.01) 0.15(0.13+0.01) -28.6% 0.15(0.14+0.01) -28.6% 0.01(0.01+0.00) -95.2% 2000.41: git diff --staged (sparse-v4) 0.22(0.21+0.01) 0.14(0.11+0.03) -36.4% 0.15(0.13+0.02) -31.8% 0.01(0.01+0.00) -95.5% ``` The `git diff` improvements are measurable. ``` 2000.42: git sparse-checkout reapply (full-v3) 0.63(0.54+0.05) 0.56(0.48+0.04) -11.1% 0.57(0.48+0.03) -9.5% 0.59(0.48+0.05) -6.3% 2000.43: git sparse-checkout reapply (full-v4) 0.60(0.54+0.02) 0.51(0.46+0.03) -15.0% 0.54(0.48+0.02) -10.0% 0.50(0.44+0.04) -16.7% 2000.44: git sparse-checkout reapply (sparse-v3) 0.91(0.86+0.05) 0.05(0.05+0.00) -94.5% 0.06(0.05+0.01) -93.4% 0.06(0.06+0.00) -93.4% 2000.45: git sparse-checkout reapply (sparse-v4) 0.92(0.88+0.04) 0.05(0.05+0.00) -94.6% 0.05(0.05+0.01) -94.6% 0.05(0.04+0.01) -94.6% ``` Finally, the `git sparse-checkout` measurements are also present. This test script is particularly valuable when contributing changes upstream. It can be good to start by adding the lines to the performance test in an early commit, then demonstrating the performance change by copying the necessary lines from the output table into your commit message.

Sparse index: integrate with `git update-index`

…tests One thing I forgot when talking about the sparse index is that we have a performance test: `t/perf/p2000-sparse-operations.sh`. This test wasn't helpful for commands like `git merge` that need a particular set of input, but work for more read-only operations. Here is a quick demonstration of how this performance test works so we could have a definitive measure of how your previous updates improved performance. To get these results, I ran the following command in `t/perf`: ``` ./run 4bcd533 f9255a5 f28fc01 b713582 -- p2000-sparse-operations.sh ``` The short-shas correspond to the merge commits for these PRs: * microsoft#410 * microsoft#421 * microsoft#417 * microsoft#419 The test takes a copy of the Git repository and creates several copies within a nested directory heirarchy. ``` Test 4bcd533 f9255a5 f28fc01 b713582 ------------------------------------------------------------------------------------------------------------------------------------------------- 2000.2: git status (full-v3) 0.19(0.15+0.05) 0.19(0.16+0.05) +0.0% 0.20(0.18+0.03) +5.3% 0.19(0.17+0.04) +0.0% 2000.3: git status (full-v4) 0.20(0.18+0.04) 0.19(0.15+0.06) -5.0% 0.21(0.18+0.05) +5.0% 0.18(0.18+0.02) -10.0% 2000.4: git status (sparse-v3) 0.04(0.04+0.04) 0.05(0.07+0.04) +25.0% 0.04(0.04+0.05) +0.0% 0.04(0.06+0.04) +0.0% 2000.5: git status (sparse-v4) 0.04(0.03+0.06) 0.04(0.05+0.05) +0.0% 0.05(0.05+0.04) +25.0% 0.05(0.06+0.04) +25.0% 2000.6: git add -A (full-v3) 0.36(0.29+0.05) 0.38(0.28+0.07) +5.6% 0.36(0.31+0.05) +0.0% 0.37(0.31+0.05) +2.8% 2000.7: git add -A (full-v4) 0.34(0.27+0.06) 0.34(0.29+0.05) +0.0% 0.34(0.29+0.04) +0.0% 0.35(0.28+0.06) +2.9% 2000.8: git add -A (sparse-v3) 0.06(0.07+0.04) 0.06(0.05+0.06) +0.0% 0.06(0.09+0.01) +0.0% 0.06(0.08+0.03) +0.0% 2000.9: git add -A (sparse-v4) 0.05(0.05+0.04) 0.05(0.05+0.07) +0.0% 0.05(0.04+0.06) +0.0% 0.06(0.06+0.05) +20.0% 2000.10: git add . (full-v3) 0.38(0.31+0.05) 0.37(0.29+0.06) -2.6% 0.37(0.30+0.07) -2.6% 0.37(0.29+0.06) -2.6% 2000.11: git add . (full-v4) 0.35(0.31+0.04) 0.35(0.29+0.07) +0.0% 0.35(0.29+0.05) +0.0% 0.34(0.29+0.06) -2.9% 2000.12: git add . (sparse-v3) 0.06(0.06+0.05) 0.06(0.05+0.06) +0.0% 0.06(0.07+0.05) +0.0% 0.06(0.09+0.03) +0.0% 2000.13: git add . (sparse-v4) 0.06(0.06+0.06) 0.06(0.07+0.04) +0.0% 0.05(0.06+0.05) -16.7% 0.05(0.05+0.07) -16.7% 2000.14: git commit -a -m A (full-v3) 0.48(0.37+0.08) 0.45(0.36+0.08) -6.2% 0.45(0.35+0.09) -6.2% 0.44(0.36+0.07) -8.3% 2000.15: git commit -a -m A (full-v4) 0.45(0.40+0.06) 0.43(0.34+0.07) -4.4% 0.45(0.37+0.06) +0.0% 0.42(0.36+0.05) -6.7% 2000.16: git commit -a -m A (sparse-v3) 0.05(0.05+0.06) 0.05(0.05+0.03) +0.0% 0.05(0.06+0.06) +0.0% 0.05(0.04+0.06) +0.0% 2000.17: git commit -a -m A (sparse-v4) 0.05(0.06+0.03) 0.05(0.06+0.04) +0.0% 0.06(0.07+0.05) +20.0% 0.05(0.04+0.06) +0.0% 2000.18: git checkout -f - (full-v3) 0.55(0.43+0.08) 0.54(0.46+0.05) -1.8% 0.55(0.46+0.07) +0.0% 0.54(0.40+0.10) -1.8% 2000.19: git checkout -f - (full-v4) 0.55(0.41+0.09) 0.50(0.40+0.09) -9.1% 0.51(0.46+0.05) -7.3% 0.51(0.44+0.06) -7.3% 2000.20: git checkout -f - (sparse-v3) 0.06(0.09+0.03) 0.06(0.08+0.03) +0.0% 0.06(0.06+0.05) +0.0% 0.07(0.09+0.03) +16.7% 2000.21: git checkout -f - (sparse-v4) 0.06(0.08+0.04) 0.05(0.07+0.05) -16.7% 0.05(0.07+0.04) -16.7% 0.06(0.09+0.03) +0.0% ``` All of the above were already integrated. ``` 2000.22: git reset (full-v3) 0.41(0.32+0.06) 0.40(0.31+0.06) -2.4% 0.41(0.33+0.05) +0.0% 0.42(0.34+0.04) +2.4% 2000.23: git reset (full-v4) 0.37(0.32+0.05) 0.35(0.30+0.05) -5.4% 0.37(0.30+0.05) +0.0% 0.35(0.31+0.03) -5.4% 2000.24: git reset (sparse-v3) 0.68(0.65+0.05) 0.55(0.52+0.04) -19.1% 0.04(0.05+0.04) -94.1% 0.04(0.05+0.04) -94.1% 2000.25: git reset (sparse-v4) 0.70(0.65+0.05) 0.54(0.50+0.06) -22.9% 0.04(0.07+0.01) -94.3% 0.03(0.05+0.05) -95.7% 2000.26: git reset --hard (full-v3) 0.54(0.43+0.07) 0.53(0.43+0.06) -1.9% 0.55(0.46+0.05) +1.9% 0.55(0.44+0.06) +1.9% 2000.27: git reset --hard (full-v4) 0.50(0.45+0.03) 0.50(0.43+0.05) +0.0% 0.49(0.41+0.06) -2.0% 0.50(0.42+0.05) +0.0% 2000.28: git reset --hard (sparse-v3) 0.83(0.76+0.06) 0.68(0.62+0.05) -18.1% 0.07(0.05+0.02) -91.6% 0.07(0.05+0.02) -91.6% 2000.29: git reset --hard (sparse-v4) 0.80(0.75+0.05) 0.69(0.62+0.06) -13.8% 0.07(0.04+0.02) -91.2% 0.07(0.04+0.03) -91.2% ``` As expected, `git reset [--hard]` improves with the sparse index integration, but remains constant across the full index case. ``` 2000.30: git update-index --add --remove (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.01+0.01) +0.0% 2000.31: git update-index --add --remove (full-v4) 0.03(0.02+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.03+0.00) +0.0% 0.03(0.02+0.01) +0.0% 2000.32: git update-index --add --remove (sparse-v3) 0.57(0.54+0.02) 0.43(0.42+0.00) -24.6% 0.44(0.41+0.03) -22.8% 0.44(0.42+0.01) -22.8% 2000.33: git update-index --add --remove (sparse-v4) 0.56(0.52+0.04) 0.43(0.42+0.01) -23.2% 0.44(0.42+0.02) -21.4% 0.42(0.41+0.01) -25.0% ``` These do not change significantly because microsoft#423 is not merged. ``` 2000.34: git diff (full-v3) 0.07(0.05+0.03) 0.06(0.05+0.03) -14.3% 0.07(0.05+0.03) +0.0% 0.06(0.05+0.03) -14.3% 2000.35: git diff (full-v4) 0.06(0.05+0.03) 0.06(0.05+0.02) +0.0% 0.06(0.05+0.02) +0.0% 0.06(0.06+0.02) +0.0% 2000.36: git diff (sparse-v3) 0.25(0.23+0.03) 0.17(0.17+0.02) -32.0% 0.18(0.18+0.02) -28.0% 0.01(0.03+0.03) -96.0% 2000.37: git diff (sparse-v4) 0.25(0.22+0.05) 0.16(0.16+0.01) -36.0% 0.18(0.15+0.04) -28.0% 0.01(0.04+0.02) -96.0% 2000.38: git diff --staged (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.02+0.00) +0.0% 2000.39: git diff --staged (full-v4) 0.04(0.03+0.01) 0.03(0.02+0.01) -25.0% 0.03(0.03+0.00) -25.0% 0.03(0.03+0.00) -25.0% 2000.40: git diff --staged (sparse-v3) 0.21(0.19+0.01) 0.15(0.13+0.01) -28.6% 0.15(0.14+0.01) -28.6% 0.01(0.01+0.00) -95.2% 2000.41: git diff --staged (sparse-v4) 0.22(0.21+0.01) 0.14(0.11+0.03) -36.4% 0.15(0.13+0.02) -31.8% 0.01(0.01+0.00) -95.5% ``` The `git diff` improvements are measurable. ``` 2000.42: git sparse-checkout reapply (full-v3) 0.63(0.54+0.05) 0.56(0.48+0.04) -11.1% 0.57(0.48+0.03) -9.5% 0.59(0.48+0.05) -6.3% 2000.43: git sparse-checkout reapply (full-v4) 0.60(0.54+0.02) 0.51(0.46+0.03) -15.0% 0.54(0.48+0.02) -10.0% 0.50(0.44+0.04) -16.7% 2000.44: git sparse-checkout reapply (sparse-v3) 0.91(0.86+0.05) 0.05(0.05+0.00) -94.5% 0.06(0.05+0.01) -93.4% 0.06(0.06+0.00) -93.4% 2000.45: git sparse-checkout reapply (sparse-v4) 0.92(0.88+0.04) 0.05(0.05+0.00) -94.6% 0.05(0.05+0.01) -94.6% 0.05(0.04+0.01) -94.6% ``` Finally, the `git sparse-checkout` measurements are also present. This test script is particularly valuable when contributing changes upstream. It can be good to start by adding the lines to the performance test in an early commit, then demonstrating the performance change by copying the necessary lines from the output table into your commit message.

Sparse index: integrate with `git update-index`

…tests One thing I forgot when talking about the sparse index is that we have a performance test: `t/perf/p2000-sparse-operations.sh`. This test wasn't helpful for commands like `git merge` that need a particular set of input, but work for more read-only operations. Here is a quick demonstration of how this performance test works so we could have a definitive measure of how your previous updates improved performance. To get these results, I ran the following command in `t/perf`: ``` ./run 4bcd533 f9255a5 f28fc01 b713582 -- p2000-sparse-operations.sh ``` The short-shas correspond to the merge commits for these PRs: * microsoft#410 * microsoft#421 * microsoft#417 * microsoft#419 The test takes a copy of the Git repository and creates several copies within a nested directory heirarchy. ``` Test 4bcd533 f9255a5 f28fc01 b713582 ------------------------------------------------------------------------------------------------------------------------------------------------- 2000.2: git status (full-v3) 0.19(0.15+0.05) 0.19(0.16+0.05) +0.0% 0.20(0.18+0.03) +5.3% 0.19(0.17+0.04) +0.0% 2000.3: git status (full-v4) 0.20(0.18+0.04) 0.19(0.15+0.06) -5.0% 0.21(0.18+0.05) +5.0% 0.18(0.18+0.02) -10.0% 2000.4: git status (sparse-v3) 0.04(0.04+0.04) 0.05(0.07+0.04) +25.0% 0.04(0.04+0.05) +0.0% 0.04(0.06+0.04) +0.0% 2000.5: git status (sparse-v4) 0.04(0.03+0.06) 0.04(0.05+0.05) +0.0% 0.05(0.05+0.04) +25.0% 0.05(0.06+0.04) +25.0% 2000.6: git add -A (full-v3) 0.36(0.29+0.05) 0.38(0.28+0.07) +5.6% 0.36(0.31+0.05) +0.0% 0.37(0.31+0.05) +2.8% 2000.7: git add -A (full-v4) 0.34(0.27+0.06) 0.34(0.29+0.05) +0.0% 0.34(0.29+0.04) +0.0% 0.35(0.28+0.06) +2.9% 2000.8: git add -A (sparse-v3) 0.06(0.07+0.04) 0.06(0.05+0.06) +0.0% 0.06(0.09+0.01) +0.0% 0.06(0.08+0.03) +0.0% 2000.9: git add -A (sparse-v4) 0.05(0.05+0.04) 0.05(0.05+0.07) +0.0% 0.05(0.04+0.06) +0.0% 0.06(0.06+0.05) +20.0% 2000.10: git add . (full-v3) 0.38(0.31+0.05) 0.37(0.29+0.06) -2.6% 0.37(0.30+0.07) -2.6% 0.37(0.29+0.06) -2.6% 2000.11: git add . (full-v4) 0.35(0.31+0.04) 0.35(0.29+0.07) +0.0% 0.35(0.29+0.05) +0.0% 0.34(0.29+0.06) -2.9% 2000.12: git add . (sparse-v3) 0.06(0.06+0.05) 0.06(0.05+0.06) +0.0% 0.06(0.07+0.05) +0.0% 0.06(0.09+0.03) +0.0% 2000.13: git add . (sparse-v4) 0.06(0.06+0.06) 0.06(0.07+0.04) +0.0% 0.05(0.06+0.05) -16.7% 0.05(0.05+0.07) -16.7% 2000.14: git commit -a -m A (full-v3) 0.48(0.37+0.08) 0.45(0.36+0.08) -6.2% 0.45(0.35+0.09) -6.2% 0.44(0.36+0.07) -8.3% 2000.15: git commit -a -m A (full-v4) 0.45(0.40+0.06) 0.43(0.34+0.07) -4.4% 0.45(0.37+0.06) +0.0% 0.42(0.36+0.05) -6.7% 2000.16: git commit -a -m A (sparse-v3) 0.05(0.05+0.06) 0.05(0.05+0.03) +0.0% 0.05(0.06+0.06) +0.0% 0.05(0.04+0.06) +0.0% 2000.17: git commit -a -m A (sparse-v4) 0.05(0.06+0.03) 0.05(0.06+0.04) +0.0% 0.06(0.07+0.05) +20.0% 0.05(0.04+0.06) +0.0% 2000.18: git checkout -f - (full-v3) 0.55(0.43+0.08) 0.54(0.46+0.05) -1.8% 0.55(0.46+0.07) +0.0% 0.54(0.40+0.10) -1.8% 2000.19: git checkout -f - (full-v4) 0.55(0.41+0.09) 0.50(0.40+0.09) -9.1% 0.51(0.46+0.05) -7.3% 0.51(0.44+0.06) -7.3% 2000.20: git checkout -f - (sparse-v3) 0.06(0.09+0.03) 0.06(0.08+0.03) +0.0% 0.06(0.06+0.05) +0.0% 0.07(0.09+0.03) +16.7% 2000.21: git checkout -f - (sparse-v4) 0.06(0.08+0.04) 0.05(0.07+0.05) -16.7% 0.05(0.07+0.04) -16.7% 0.06(0.09+0.03) +0.0% ``` All of the above were already integrated. ``` 2000.22: git reset (full-v3) 0.41(0.32+0.06) 0.40(0.31+0.06) -2.4% 0.41(0.33+0.05) +0.0% 0.42(0.34+0.04) +2.4% 2000.23: git reset (full-v4) 0.37(0.32+0.05) 0.35(0.30+0.05) -5.4% 0.37(0.30+0.05) +0.0% 0.35(0.31+0.03) -5.4% 2000.24: git reset (sparse-v3) 0.68(0.65+0.05) 0.55(0.52+0.04) -19.1% 0.04(0.05+0.04) -94.1% 0.04(0.05+0.04) -94.1% 2000.25: git reset (sparse-v4) 0.70(0.65+0.05) 0.54(0.50+0.06) -22.9% 0.04(0.07+0.01) -94.3% 0.03(0.05+0.05) -95.7% 2000.26: git reset --hard (full-v3) 0.54(0.43+0.07) 0.53(0.43+0.06) -1.9% 0.55(0.46+0.05) +1.9% 0.55(0.44+0.06) +1.9% 2000.27: git reset --hard (full-v4) 0.50(0.45+0.03) 0.50(0.43+0.05) +0.0% 0.49(0.41+0.06) -2.0% 0.50(0.42+0.05) +0.0% 2000.28: git reset --hard (sparse-v3) 0.83(0.76+0.06) 0.68(0.62+0.05) -18.1% 0.07(0.05+0.02) -91.6% 0.07(0.05+0.02) -91.6% 2000.29: git reset --hard (sparse-v4) 0.80(0.75+0.05) 0.69(0.62+0.06) -13.8% 0.07(0.04+0.02) -91.2% 0.07(0.04+0.03) -91.2% ``` As expected, `git reset [--hard]` improves with the sparse index integration, but remains constant across the full index case. ``` 2000.30: git update-index --add --remove (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.01+0.01) +0.0% 2000.31: git update-index --add --remove (full-v4) 0.03(0.02+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.03+0.00) +0.0% 0.03(0.02+0.01) +0.0% 2000.32: git update-index --add --remove (sparse-v3) 0.57(0.54+0.02) 0.43(0.42+0.00) -24.6% 0.44(0.41+0.03) -22.8% 0.44(0.42+0.01) -22.8% 2000.33: git update-index --add --remove (sparse-v4) 0.56(0.52+0.04) 0.43(0.42+0.01) -23.2% 0.44(0.42+0.02) -21.4% 0.42(0.41+0.01) -25.0% ``` These do not change significantly because microsoft#423 is not merged. ``` 2000.34: git diff (full-v3) 0.07(0.05+0.03) 0.06(0.05+0.03) -14.3% 0.07(0.05+0.03) +0.0% 0.06(0.05+0.03) -14.3% 2000.35: git diff (full-v4) 0.06(0.05+0.03) 0.06(0.05+0.02) +0.0% 0.06(0.05+0.02) +0.0% 0.06(0.06+0.02) +0.0% 2000.36: git diff (sparse-v3) 0.25(0.23+0.03) 0.17(0.17+0.02) -32.0% 0.18(0.18+0.02) -28.0% 0.01(0.03+0.03) -96.0% 2000.37: git diff (sparse-v4) 0.25(0.22+0.05) 0.16(0.16+0.01) -36.0% 0.18(0.15+0.04) -28.0% 0.01(0.04+0.02) -96.0% 2000.38: git diff --staged (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.02+0.00) +0.0% 2000.39: git diff --staged (full-v4) 0.04(0.03+0.01) 0.03(0.02+0.01) -25.0% 0.03(0.03+0.00) -25.0% 0.03(0.03+0.00) -25.0% 2000.40: git diff --staged (sparse-v3) 0.21(0.19+0.01) 0.15(0.13+0.01) -28.6% 0.15(0.14+0.01) -28.6% 0.01(0.01+0.00) -95.2% 2000.41: git diff --staged (sparse-v4) 0.22(0.21+0.01) 0.14(0.11+0.03) -36.4% 0.15(0.13+0.02) -31.8% 0.01(0.01+0.00) -95.5% ``` The `git diff` improvements are measurable. ``` 2000.42: git sparse-checkout reapply (full-v3) 0.63(0.54+0.05) 0.56(0.48+0.04) -11.1% 0.57(0.48+0.03) -9.5% 0.59(0.48+0.05) -6.3% 2000.43: git sparse-checkout reapply (full-v4) 0.60(0.54+0.02) 0.51(0.46+0.03) -15.0% 0.54(0.48+0.02) -10.0% 0.50(0.44+0.04) -16.7% 2000.44: git sparse-checkout reapply (sparse-v3) 0.91(0.86+0.05) 0.05(0.05+0.00) -94.5% 0.06(0.05+0.01) -93.4% 0.06(0.06+0.00) -93.4% 2000.45: git sparse-checkout reapply (sparse-v4) 0.92(0.88+0.04) 0.05(0.05+0.00) -94.6% 0.05(0.05+0.01) -94.6% 0.05(0.04+0.01) -94.6% ``` Finally, the `git sparse-checkout` measurements are also present. This test script is particularly valuable when contributing changes upstream. It can be good to start by adding the lines to the performance test in an early commit, then demonstrating the performance change by copying the necessary lines from the output table into your commit message.

Sparse index: integrate with `git update-index`

…tests One thing I forgot when talking about the sparse index is that we have a performance test: `t/perf/p2000-sparse-operations.sh`. This test wasn't helpful for commands like `git merge` that need a particular set of input, but work for more read-only operations. Here is a quick demonstration of how this performance test works so we could have a definitive measure of how your previous updates improved performance. To get these results, I ran the following command in `t/perf`: ``` ./run 4bcd533 f9255a5 f28fc01 b713582 -- p2000-sparse-operations.sh ``` The short-shas correspond to the merge commits for these PRs: * microsoft#410 * microsoft#421 * microsoft#417 * microsoft#419 The test takes a copy of the Git repository and creates several copies within a nested directory heirarchy. ``` Test 4bcd533 f9255a5 f28fc01 b713582 ------------------------------------------------------------------------------------------------------------------------------------------------- 2000.2: git status (full-v3) 0.19(0.15+0.05) 0.19(0.16+0.05) +0.0% 0.20(0.18+0.03) +5.3% 0.19(0.17+0.04) +0.0% 2000.3: git status (full-v4) 0.20(0.18+0.04) 0.19(0.15+0.06) -5.0% 0.21(0.18+0.05) +5.0% 0.18(0.18+0.02) -10.0% 2000.4: git status (sparse-v3) 0.04(0.04+0.04) 0.05(0.07+0.04) +25.0% 0.04(0.04+0.05) +0.0% 0.04(0.06+0.04) +0.0% 2000.5: git status (sparse-v4) 0.04(0.03+0.06) 0.04(0.05+0.05) +0.0% 0.05(0.05+0.04) +25.0% 0.05(0.06+0.04) +25.0% 2000.6: git add -A (full-v3) 0.36(0.29+0.05) 0.38(0.28+0.07) +5.6% 0.36(0.31+0.05) +0.0% 0.37(0.31+0.05) +2.8% 2000.7: git add -A (full-v4) 0.34(0.27+0.06) 0.34(0.29+0.05) +0.0% 0.34(0.29+0.04) +0.0% 0.35(0.28+0.06) +2.9% 2000.8: git add -A (sparse-v3) 0.06(0.07+0.04) 0.06(0.05+0.06) +0.0% 0.06(0.09+0.01) +0.0% 0.06(0.08+0.03) +0.0% 2000.9: git add -A (sparse-v4) 0.05(0.05+0.04) 0.05(0.05+0.07) +0.0% 0.05(0.04+0.06) +0.0% 0.06(0.06+0.05) +20.0% 2000.10: git add . (full-v3) 0.38(0.31+0.05) 0.37(0.29+0.06) -2.6% 0.37(0.30+0.07) -2.6% 0.37(0.29+0.06) -2.6% 2000.11: git add . (full-v4) 0.35(0.31+0.04) 0.35(0.29+0.07) +0.0% 0.35(0.29+0.05) +0.0% 0.34(0.29+0.06) -2.9% 2000.12: git add . (sparse-v3) 0.06(0.06+0.05) 0.06(0.05+0.06) +0.0% 0.06(0.07+0.05) +0.0% 0.06(0.09+0.03) +0.0% 2000.13: git add . (sparse-v4) 0.06(0.06+0.06) 0.06(0.07+0.04) +0.0% 0.05(0.06+0.05) -16.7% 0.05(0.05+0.07) -16.7% 2000.14: git commit -a -m A (full-v3) 0.48(0.37+0.08) 0.45(0.36+0.08) -6.2% 0.45(0.35+0.09) -6.2% 0.44(0.36+0.07) -8.3% 2000.15: git commit -a -m A (full-v4) 0.45(0.40+0.06) 0.43(0.34+0.07) -4.4% 0.45(0.37+0.06) +0.0% 0.42(0.36+0.05) -6.7% 2000.16: git commit -a -m A (sparse-v3) 0.05(0.05+0.06) 0.05(0.05+0.03) +0.0% 0.05(0.06+0.06) +0.0% 0.05(0.04+0.06) +0.0% 2000.17: git commit -a -m A (sparse-v4) 0.05(0.06+0.03) 0.05(0.06+0.04) +0.0% 0.06(0.07+0.05) +20.0% 0.05(0.04+0.06) +0.0% 2000.18: git checkout -f - (full-v3) 0.55(0.43+0.08) 0.54(0.46+0.05) -1.8% 0.55(0.46+0.07) +0.0% 0.54(0.40+0.10) -1.8% 2000.19: git checkout -f - (full-v4) 0.55(0.41+0.09) 0.50(0.40+0.09) -9.1% 0.51(0.46+0.05) -7.3% 0.51(0.44+0.06) -7.3% 2000.20: git checkout -f - (sparse-v3) 0.06(0.09+0.03) 0.06(0.08+0.03) +0.0% 0.06(0.06+0.05) +0.0% 0.07(0.09+0.03) +16.7% 2000.21: git checkout -f - (sparse-v4) 0.06(0.08+0.04) 0.05(0.07+0.05) -16.7% 0.05(0.07+0.04) -16.7% 0.06(0.09+0.03) +0.0% ``` All of the above were already integrated. ``` 2000.22: git reset (full-v3) 0.41(0.32+0.06) 0.40(0.31+0.06) -2.4% 0.41(0.33+0.05) +0.0% 0.42(0.34+0.04) +2.4% 2000.23: git reset (full-v4) 0.37(0.32+0.05) 0.35(0.30+0.05) -5.4% 0.37(0.30+0.05) +0.0% 0.35(0.31+0.03) -5.4% 2000.24: git reset (sparse-v3) 0.68(0.65+0.05) 0.55(0.52+0.04) -19.1% 0.04(0.05+0.04) -94.1% 0.04(0.05+0.04) -94.1% 2000.25: git reset (sparse-v4) 0.70(0.65+0.05) 0.54(0.50+0.06) -22.9% 0.04(0.07+0.01) -94.3% 0.03(0.05+0.05) -95.7% 2000.26: git reset --hard (full-v3) 0.54(0.43+0.07) 0.53(0.43+0.06) -1.9% 0.55(0.46+0.05) +1.9% 0.55(0.44+0.06) +1.9% 2000.27: git reset --hard (full-v4) 0.50(0.45+0.03) 0.50(0.43+0.05) +0.0% 0.49(0.41+0.06) -2.0% 0.50(0.42+0.05) +0.0% 2000.28: git reset --hard (sparse-v3) 0.83(0.76+0.06) 0.68(0.62+0.05) -18.1% 0.07(0.05+0.02) -91.6% 0.07(0.05+0.02) -91.6% 2000.29: git reset --hard (sparse-v4) 0.80(0.75+0.05) 0.69(0.62+0.06) -13.8% 0.07(0.04+0.02) -91.2% 0.07(0.04+0.03) -91.2% ``` As expected, `git reset [--hard]` improves with the sparse index integration, but remains constant across the full index case. ``` 2000.30: git update-index --add --remove (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.01+0.01) +0.0% 2000.31: git update-index --add --remove (full-v4) 0.03(0.02+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.03+0.00) +0.0% 0.03(0.02+0.01) +0.0% 2000.32: git update-index --add --remove (sparse-v3) 0.57(0.54+0.02) 0.43(0.42+0.00) -24.6% 0.44(0.41+0.03) -22.8% 0.44(0.42+0.01) -22.8% 2000.33: git update-index --add --remove (sparse-v4) 0.56(0.52+0.04) 0.43(0.42+0.01) -23.2% 0.44(0.42+0.02) -21.4% 0.42(0.41+0.01) -25.0% ``` These do not change significantly because microsoft#423 is not merged. ``` 2000.34: git diff (full-v3) 0.07(0.05+0.03) 0.06(0.05+0.03) -14.3% 0.07(0.05+0.03) +0.0% 0.06(0.05+0.03) -14.3% 2000.35: git diff (full-v4) 0.06(0.05+0.03) 0.06(0.05+0.02) +0.0% 0.06(0.05+0.02) +0.0% 0.06(0.06+0.02) +0.0% 2000.36: git diff (sparse-v3) 0.25(0.23+0.03) 0.17(0.17+0.02) -32.0% 0.18(0.18+0.02) -28.0% 0.01(0.03+0.03) -96.0% 2000.37: git diff (sparse-v4) 0.25(0.22+0.05) 0.16(0.16+0.01) -36.0% 0.18(0.15+0.04) -28.0% 0.01(0.04+0.02) -96.0% 2000.38: git diff --staged (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.02+0.00) +0.0% 2000.39: git diff --staged (full-v4) 0.04(0.03+0.01) 0.03(0.02+0.01) -25.0% 0.03(0.03+0.00) -25.0% 0.03(0.03+0.00) -25.0% 2000.40: git diff --staged (sparse-v3) 0.21(0.19+0.01) 0.15(0.13+0.01) -28.6% 0.15(0.14+0.01) -28.6% 0.01(0.01+0.00) -95.2% 2000.41: git diff --staged (sparse-v4) 0.22(0.21+0.01) 0.14(0.11+0.03) -36.4% 0.15(0.13+0.02) -31.8% 0.01(0.01+0.00) -95.5% ``` The `git diff` improvements are measurable. ``` 2000.42: git sparse-checkout reapply (full-v3) 0.63(0.54+0.05) 0.56(0.48+0.04) -11.1% 0.57(0.48+0.03) -9.5% 0.59(0.48+0.05) -6.3% 2000.43: git sparse-checkout reapply (full-v4) 0.60(0.54+0.02) 0.51(0.46+0.03) -15.0% 0.54(0.48+0.02) -10.0% 0.50(0.44+0.04) -16.7% 2000.44: git sparse-checkout reapply (sparse-v3) 0.91(0.86+0.05) 0.05(0.05+0.00) -94.5% 0.06(0.05+0.01) -93.4% 0.06(0.06+0.00) -93.4% 2000.45: git sparse-checkout reapply (sparse-v4) 0.92(0.88+0.04) 0.05(0.05+0.00) -94.6% 0.05(0.05+0.01) -94.6% 0.05(0.04+0.01) -94.6% ``` Finally, the `git sparse-checkout` measurements are also present. This test script is particularly valuable when contributing changes upstream. It can be good to start by adding the lines to the performance test in an early commit, then demonstrating the performance change by copying the necessary lines from the output table into your commit message.

Sparse index: integrate with `git update-index`

…tests One thing I forgot when talking about the sparse index is that we have a performance test: `t/perf/p2000-sparse-operations.sh`. This test wasn't helpful for commands like `git merge` that need a particular set of input, but work for more read-only operations. Here is a quick demonstration of how this performance test works so we could have a definitive measure of how your previous updates improved performance. To get these results, I ran the following command in `t/perf`: ``` ./run 4bcd533 f9255a5 f28fc01 b713582 -- p2000-sparse-operations.sh ``` The short-shas correspond to the merge commits for these PRs: * microsoft#410 * microsoft#421 * microsoft#417 * microsoft#419 The test takes a copy of the Git repository and creates several copies within a nested directory heirarchy. ``` Test 4bcd533 f9255a5 f28fc01 b713582 ------------------------------------------------------------------------------------------------------------------------------------------------- 2000.2: git status (full-v3) 0.19(0.15+0.05) 0.19(0.16+0.05) +0.0% 0.20(0.18+0.03) +5.3% 0.19(0.17+0.04) +0.0% 2000.3: git status (full-v4) 0.20(0.18+0.04) 0.19(0.15+0.06) -5.0% 0.21(0.18+0.05) +5.0% 0.18(0.18+0.02) -10.0% 2000.4: git status (sparse-v3) 0.04(0.04+0.04) 0.05(0.07+0.04) +25.0% 0.04(0.04+0.05) +0.0% 0.04(0.06+0.04) +0.0% 2000.5: git status (sparse-v4) 0.04(0.03+0.06) 0.04(0.05+0.05) +0.0% 0.05(0.05+0.04) +25.0% 0.05(0.06+0.04) +25.0% 2000.6: git add -A (full-v3) 0.36(0.29+0.05) 0.38(0.28+0.07) +5.6% 0.36(0.31+0.05) +0.0% 0.37(0.31+0.05) +2.8% 2000.7: git add -A (full-v4) 0.34(0.27+0.06) 0.34(0.29+0.05) +0.0% 0.34(0.29+0.04) +0.0% 0.35(0.28+0.06) +2.9% 2000.8: git add -A (sparse-v3) 0.06(0.07+0.04) 0.06(0.05+0.06) +0.0% 0.06(0.09+0.01) +0.0% 0.06(0.08+0.03) +0.0% 2000.9: git add -A (sparse-v4) 0.05(0.05+0.04) 0.05(0.05+0.07) +0.0% 0.05(0.04+0.06) +0.0% 0.06(0.06+0.05) +20.0% 2000.10: git add . (full-v3) 0.38(0.31+0.05) 0.37(0.29+0.06) -2.6% 0.37(0.30+0.07) -2.6% 0.37(0.29+0.06) -2.6% 2000.11: git add . (full-v4) 0.35(0.31+0.04) 0.35(0.29+0.07) +0.0% 0.35(0.29+0.05) +0.0% 0.34(0.29+0.06) -2.9% 2000.12: git add . (sparse-v3) 0.06(0.06+0.05) 0.06(0.05+0.06) +0.0% 0.06(0.07+0.05) +0.0% 0.06(0.09+0.03) +0.0% 2000.13: git add . (sparse-v4) 0.06(0.06+0.06) 0.06(0.07+0.04) +0.0% 0.05(0.06+0.05) -16.7% 0.05(0.05+0.07) -16.7% 2000.14: git commit -a -m A (full-v3) 0.48(0.37+0.08) 0.45(0.36+0.08) -6.2% 0.45(0.35+0.09) -6.2% 0.44(0.36+0.07) -8.3% 2000.15: git commit -a -m A (full-v4) 0.45(0.40+0.06) 0.43(0.34+0.07) -4.4% 0.45(0.37+0.06) +0.0% 0.42(0.36+0.05) -6.7% 2000.16: git commit -a -m A (sparse-v3) 0.05(0.05+0.06) 0.05(0.05+0.03) +0.0% 0.05(0.06+0.06) +0.0% 0.05(0.04+0.06) +0.0% 2000.17: git commit -a -m A (sparse-v4) 0.05(0.06+0.03) 0.05(0.06+0.04) +0.0% 0.06(0.07+0.05) +20.0% 0.05(0.04+0.06) +0.0% 2000.18: git checkout -f - (full-v3) 0.55(0.43+0.08) 0.54(0.46+0.05) -1.8% 0.55(0.46+0.07) +0.0% 0.54(0.40+0.10) -1.8% 2000.19: git checkout -f - (full-v4) 0.55(0.41+0.09) 0.50(0.40+0.09) -9.1% 0.51(0.46+0.05) -7.3% 0.51(0.44+0.06) -7.3% 2000.20: git checkout -f - (sparse-v3) 0.06(0.09+0.03) 0.06(0.08+0.03) +0.0% 0.06(0.06+0.05) +0.0% 0.07(0.09+0.03) +16.7% 2000.21: git checkout -f - (sparse-v4) 0.06(0.08+0.04) 0.05(0.07+0.05) -16.7% 0.05(0.07+0.04) -16.7% 0.06(0.09+0.03) +0.0% ``` All of the above were already integrated. ``` 2000.22: git reset (full-v3) 0.41(0.32+0.06) 0.40(0.31+0.06) -2.4% 0.41(0.33+0.05) +0.0% 0.42(0.34+0.04) +2.4% 2000.23: git reset (full-v4) 0.37(0.32+0.05) 0.35(0.30+0.05) -5.4% 0.37(0.30+0.05) +0.0% 0.35(0.31+0.03) -5.4% 2000.24: git reset (sparse-v3) 0.68(0.65+0.05) 0.55(0.52+0.04) -19.1% 0.04(0.05+0.04) -94.1% 0.04(0.05+0.04) -94.1% 2000.25: git reset (sparse-v4) 0.70(0.65+0.05) 0.54(0.50+0.06) -22.9% 0.04(0.07+0.01) -94.3% 0.03(0.05+0.05) -95.7% 2000.26: git reset --hard (full-v3) 0.54(0.43+0.07) 0.53(0.43+0.06) -1.9% 0.55(0.46+0.05) +1.9% 0.55(0.44+0.06) +1.9% 2000.27: git reset --hard (full-v4) 0.50(0.45+0.03) 0.50(0.43+0.05) +0.0% 0.49(0.41+0.06) -2.0% 0.50(0.42+0.05) +0.0% 2000.28: git reset --hard (sparse-v3) 0.83(0.76+0.06) 0.68(0.62+0.05) -18.1% 0.07(0.05+0.02) -91.6% 0.07(0.05+0.02) -91.6% 2000.29: git reset --hard (sparse-v4) 0.80(0.75+0.05) 0.69(0.62+0.06) -13.8% 0.07(0.04+0.02) -91.2% 0.07(0.04+0.03) -91.2% ``` As expected, `git reset [--hard]` improves with the sparse index integration, but remains constant across the full index case. ``` 2000.30: git update-index --add --remove (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.01+0.01) +0.0% 2000.31: git update-index --add --remove (full-v4) 0.03(0.02+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.03+0.00) +0.0% 0.03(0.02+0.01) +0.0% 2000.32: git update-index --add --remove (sparse-v3) 0.57(0.54+0.02) 0.43(0.42+0.00) -24.6% 0.44(0.41+0.03) -22.8% 0.44(0.42+0.01) -22.8% 2000.33: git update-index --add --remove (sparse-v4) 0.56(0.52+0.04) 0.43(0.42+0.01) -23.2% 0.44(0.42+0.02) -21.4% 0.42(0.41+0.01) -25.0% ``` These do not change significantly because microsoft#423 is not merged. ``` 2000.34: git diff (full-v3) 0.07(0.05+0.03) 0.06(0.05+0.03) -14.3% 0.07(0.05+0.03) +0.0% 0.06(0.05+0.03) -14.3% 2000.35: git diff (full-v4) 0.06(0.05+0.03) 0.06(0.05+0.02) +0.0% 0.06(0.05+0.02) +0.0% 0.06(0.06+0.02) +0.0% 2000.36: git diff (sparse-v3) 0.25(0.23+0.03) 0.17(0.17+0.02) -32.0% 0.18(0.18+0.02) -28.0% 0.01(0.03+0.03) -96.0% 2000.37: git diff (sparse-v4) 0.25(0.22+0.05) 0.16(0.16+0.01) -36.0% 0.18(0.15+0.04) -28.0% 0.01(0.04+0.02) -96.0% 2000.38: git diff --staged (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.02+0.00) +0.0% 2000.39: git diff --staged (full-v4) 0.04(0.03+0.01) 0.03(0.02+0.01) -25.0% 0.03(0.03+0.00) -25.0% 0.03(0.03+0.00) -25.0% 2000.40: git diff --staged (sparse-v3) 0.21(0.19+0.01) 0.15(0.13+0.01) -28.6% 0.15(0.14+0.01) -28.6% 0.01(0.01+0.00) -95.2% 2000.41: git diff --staged (sparse-v4) 0.22(0.21+0.01) 0.14(0.11+0.03) -36.4% 0.15(0.13+0.02) -31.8% 0.01(0.01+0.00) -95.5% ``` The `git diff` improvements are measurable. ``` 2000.42: git sparse-checkout reapply (full-v3) 0.63(0.54+0.05) 0.56(0.48+0.04) -11.1% 0.57(0.48+0.03) -9.5% 0.59(0.48+0.05) -6.3% 2000.43: git sparse-checkout reapply (full-v4) 0.60(0.54+0.02) 0.51(0.46+0.03) -15.0% 0.54(0.48+0.02) -10.0% 0.50(0.44+0.04) -16.7% 2000.44: git sparse-checkout reapply (sparse-v3) 0.91(0.86+0.05) 0.05(0.05+0.00) -94.5% 0.06(0.05+0.01) -93.4% 0.06(0.06+0.00) -93.4% 2000.45: git sparse-checkout reapply (sparse-v4) 0.92(0.88+0.04) 0.05(0.05+0.00) -94.6% 0.05(0.05+0.01) -94.6% 0.05(0.04+0.01) -94.6% ``` Finally, the `git sparse-checkout` measurements are also present. This test script is particularly valuable when contributing changes upstream. It can be good to start by adding the lines to the performance test in an early commit, then demonstrating the performance change by copying the necessary lines from the output table into your commit message.

Sparse index: integrate with `git update-index`

…tests One thing I forgot when talking about the sparse index is that we have a performance test: `t/perf/p2000-sparse-operations.sh`. This test wasn't helpful for commands like `git merge` that need a particular set of input, but work for more read-only operations. Here is a quick demonstration of how this performance test works so we could have a definitive measure of how your previous updates improved performance. To get these results, I ran the following command in `t/perf`: ``` ./run 4bcd533 f9255a5 f28fc01 b713582 -- p2000-sparse-operations.sh ``` The short-shas correspond to the merge commits for these PRs: * microsoft#410 * microsoft#421 * microsoft#417 * microsoft#419 The test takes a copy of the Git repository and creates several copies within a nested directory heirarchy. ``` Test 4bcd533 f9255a5 f28fc01 b713582 ------------------------------------------------------------------------------------------------------------------------------------------------- 2000.2: git status (full-v3) 0.19(0.15+0.05) 0.19(0.16+0.05) +0.0% 0.20(0.18+0.03) +5.3% 0.19(0.17+0.04) +0.0% 2000.3: git status (full-v4) 0.20(0.18+0.04) 0.19(0.15+0.06) -5.0% 0.21(0.18+0.05) +5.0% 0.18(0.18+0.02) -10.0% 2000.4: git status (sparse-v3) 0.04(0.04+0.04) 0.05(0.07+0.04) +25.0% 0.04(0.04+0.05) +0.0% 0.04(0.06+0.04) +0.0% 2000.5: git status (sparse-v4) 0.04(0.03+0.06) 0.04(0.05+0.05) +0.0% 0.05(0.05+0.04) +25.0% 0.05(0.06+0.04) +25.0% 2000.6: git add -A (full-v3) 0.36(0.29+0.05) 0.38(0.28+0.07) +5.6% 0.36(0.31+0.05) +0.0% 0.37(0.31+0.05) +2.8% 2000.7: git add -A (full-v4) 0.34(0.27+0.06) 0.34(0.29+0.05) +0.0% 0.34(0.29+0.04) +0.0% 0.35(0.28+0.06) +2.9% 2000.8: git add -A (sparse-v3) 0.06(0.07+0.04) 0.06(0.05+0.06) +0.0% 0.06(0.09+0.01) +0.0% 0.06(0.08+0.03) +0.0% 2000.9: git add -A (sparse-v4) 0.05(0.05+0.04) 0.05(0.05+0.07) +0.0% 0.05(0.04+0.06) +0.0% 0.06(0.06+0.05) +20.0% 2000.10: git add . (full-v3) 0.38(0.31+0.05) 0.37(0.29+0.06) -2.6% 0.37(0.30+0.07) -2.6% 0.37(0.29+0.06) -2.6% 2000.11: git add . (full-v4) 0.35(0.31+0.04) 0.35(0.29+0.07) +0.0% 0.35(0.29+0.05) +0.0% 0.34(0.29+0.06) -2.9% 2000.12: git add . (sparse-v3) 0.06(0.06+0.05) 0.06(0.05+0.06) +0.0% 0.06(0.07+0.05) +0.0% 0.06(0.09+0.03) +0.0% 2000.13: git add . (sparse-v4) 0.06(0.06+0.06) 0.06(0.07+0.04) +0.0% 0.05(0.06+0.05) -16.7% 0.05(0.05+0.07) -16.7% 2000.14: git commit -a -m A (full-v3) 0.48(0.37+0.08) 0.45(0.36+0.08) -6.2% 0.45(0.35+0.09) -6.2% 0.44(0.36+0.07) -8.3% 2000.15: git commit -a -m A (full-v4) 0.45(0.40+0.06) 0.43(0.34+0.07) -4.4% 0.45(0.37+0.06) +0.0% 0.42(0.36+0.05) -6.7% 2000.16: git commit -a -m A (sparse-v3) 0.05(0.05+0.06) 0.05(0.05+0.03) +0.0% 0.05(0.06+0.06) +0.0% 0.05(0.04+0.06) +0.0% 2000.17: git commit -a -m A (sparse-v4) 0.05(0.06+0.03) 0.05(0.06+0.04) +0.0% 0.06(0.07+0.05) +20.0% 0.05(0.04+0.06) +0.0% 2000.18: git checkout -f - (full-v3) 0.55(0.43+0.08) 0.54(0.46+0.05) -1.8% 0.55(0.46+0.07) +0.0% 0.54(0.40+0.10) -1.8% 2000.19: git checkout -f - (full-v4) 0.55(0.41+0.09) 0.50(0.40+0.09) -9.1% 0.51(0.46+0.05) -7.3% 0.51(0.44+0.06) -7.3% 2000.20: git checkout -f - (sparse-v3) 0.06(0.09+0.03) 0.06(0.08+0.03) +0.0% 0.06(0.06+0.05) +0.0% 0.07(0.09+0.03) +16.7% 2000.21: git checkout -f - (sparse-v4) 0.06(0.08+0.04) 0.05(0.07+0.05) -16.7% 0.05(0.07+0.04) -16.7% 0.06(0.09+0.03) +0.0% ``` All of the above were already integrated. ``` 2000.22: git reset (full-v3) 0.41(0.32+0.06) 0.40(0.31+0.06) -2.4% 0.41(0.33+0.05) +0.0% 0.42(0.34+0.04) +2.4% 2000.23: git reset (full-v4) 0.37(0.32+0.05) 0.35(0.30+0.05) -5.4% 0.37(0.30+0.05) +0.0% 0.35(0.31+0.03) -5.4% 2000.24: git reset (sparse-v3) 0.68(0.65+0.05) 0.55(0.52+0.04) -19.1% 0.04(0.05+0.04) -94.1% 0.04(0.05+0.04) -94.1% 2000.25: git reset (sparse-v4) 0.70(0.65+0.05) 0.54(0.50+0.06) -22.9% 0.04(0.07+0.01) -94.3% 0.03(0.05+0.05) -95.7% 2000.26: git reset --hard (full-v3) 0.54(0.43+0.07) 0.53(0.43+0.06) -1.9% 0.55(0.46+0.05) +1.9% 0.55(0.44+0.06) +1.9% 2000.27: git reset --hard (full-v4) 0.50(0.45+0.03) 0.50(0.43+0.05) +0.0% 0.49(0.41+0.06) -2.0% 0.50(0.42+0.05) +0.0% 2000.28: git reset --hard (sparse-v3) 0.83(0.76+0.06) 0.68(0.62+0.05) -18.1% 0.07(0.05+0.02) -91.6% 0.07(0.05+0.02) -91.6% 2000.29: git reset --hard (sparse-v4) 0.80(0.75+0.05) 0.69(0.62+0.06) -13.8% 0.07(0.04+0.02) -91.2% 0.07(0.04+0.03) -91.2% ``` As expected, `git reset [--hard]` improves with the sparse index integration, but remains constant across the full index case. ``` 2000.30: git update-index --add --remove (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.01+0.01) +0.0% 2000.31: git update-index --add --remove (full-v4) 0.03(0.02+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.03+0.00) +0.0% 0.03(0.02+0.01) +0.0% 2000.32: git update-index --add --remove (sparse-v3) 0.57(0.54+0.02) 0.43(0.42+0.00) -24.6% 0.44(0.41+0.03) -22.8% 0.44(0.42+0.01) -22.8% 2000.33: git update-index --add --remove (sparse-v4) 0.56(0.52+0.04) 0.43(0.42+0.01) -23.2% 0.44(0.42+0.02) -21.4% 0.42(0.41+0.01) -25.0% ``` These do not change significantly because microsoft#423 is not merged. ``` 2000.34: git diff (full-v3) 0.07(0.05+0.03) 0.06(0.05+0.03) -14.3% 0.07(0.05+0.03) +0.0% 0.06(0.05+0.03) -14.3% 2000.35: git diff (full-v4) 0.06(0.05+0.03) 0.06(0.05+0.02) +0.0% 0.06(0.05+0.02) +0.0% 0.06(0.06+0.02) +0.0% 2000.36: git diff (sparse-v3) 0.25(0.23+0.03) 0.17(0.17+0.02) -32.0% 0.18(0.18+0.02) -28.0% 0.01(0.03+0.03) -96.0% 2000.37: git diff (sparse-v4) 0.25(0.22+0.05) 0.16(0.16+0.01) -36.0% 0.18(0.15+0.04) -28.0% 0.01(0.04+0.02) -96.0% 2000.38: git diff --staged (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.02+0.00) +0.0% 2000.39: git diff --staged (full-v4) 0.04(0.03+0.01) 0.03(0.02+0.01) -25.0% 0.03(0.03+0.00) -25.0% 0.03(0.03+0.00) -25.0% 2000.40: git diff --staged (sparse-v3) 0.21(0.19+0.01) 0.15(0.13+0.01) -28.6% 0.15(0.14+0.01) -28.6% 0.01(0.01+0.00) -95.2% 2000.41: git diff --staged (sparse-v4) 0.22(0.21+0.01) 0.14(0.11+0.03) -36.4% 0.15(0.13+0.02) -31.8% 0.01(0.01+0.00) -95.5% ``` The `git diff` improvements are measurable. ``` 2000.42: git sparse-checkout reapply (full-v3) 0.63(0.54+0.05) 0.56(0.48+0.04) -11.1% 0.57(0.48+0.03) -9.5% 0.59(0.48+0.05) -6.3% 2000.43: git sparse-checkout reapply (full-v4) 0.60(0.54+0.02) 0.51(0.46+0.03) -15.0% 0.54(0.48+0.02) -10.0% 0.50(0.44+0.04) -16.7% 2000.44: git sparse-checkout reapply (sparse-v3) 0.91(0.86+0.05) 0.05(0.05+0.00) -94.5% 0.06(0.05+0.01) -93.4% 0.06(0.06+0.00) -93.4% 2000.45: git sparse-checkout reapply (sparse-v4) 0.92(0.88+0.04) 0.05(0.05+0.00) -94.6% 0.05(0.05+0.01) -94.6% 0.05(0.04+0.01) -94.6% ``` Finally, the `git sparse-checkout` measurements are also present. This test script is particularly valuable when contributing changes upstream. It can be good to start by adding the lines to the performance test in an early commit, then demonstrating the performance change by copying the necessary lines from the output table into your commit message.

Sparse index: integrate with `git update-index`

…tests One thing I forgot when talking about the sparse index is that we have a performance test: `t/perf/p2000-sparse-operations.sh`. This test wasn't helpful for commands like `git merge` that need a particular set of input, but work for more read-only operations. Here is a quick demonstration of how this performance test works so we could have a definitive measure of how your previous updates improved performance. To get these results, I ran the following command in `t/perf`: ``` ./run 4bcd533 f9255a5 f28fc01 b713582 -- p2000-sparse-operations.sh ``` The short-shas correspond to the merge commits for these PRs: * microsoft#410 * microsoft#421 * microsoft#417 * microsoft#419 The test takes a copy of the Git repository and creates several copies within a nested directory heirarchy. ``` Test 4bcd533 f9255a5 f28fc01 b713582 ------------------------------------------------------------------------------------------------------------------------------------------------- 2000.2: git status (full-v3) 0.19(0.15+0.05) 0.19(0.16+0.05) +0.0% 0.20(0.18+0.03) +5.3% 0.19(0.17+0.04) +0.0% 2000.3: git status (full-v4) 0.20(0.18+0.04) 0.19(0.15+0.06) -5.0% 0.21(0.18+0.05) +5.0% 0.18(0.18+0.02) -10.0% 2000.4: git status (sparse-v3) 0.04(0.04+0.04) 0.05(0.07+0.04) +25.0% 0.04(0.04+0.05) +0.0% 0.04(0.06+0.04) +0.0% 2000.5: git status (sparse-v4) 0.04(0.03+0.06) 0.04(0.05+0.05) +0.0% 0.05(0.05+0.04) +25.0% 0.05(0.06+0.04) +25.0% 2000.6: git add -A (full-v3) 0.36(0.29+0.05) 0.38(0.28+0.07) +5.6% 0.36(0.31+0.05) +0.0% 0.37(0.31+0.05) +2.8% 2000.7: git add -A (full-v4) 0.34(0.27+0.06) 0.34(0.29+0.05) +0.0% 0.34(0.29+0.04) +0.0% 0.35(0.28+0.06) +2.9% 2000.8: git add -A (sparse-v3) 0.06(0.07+0.04) 0.06(0.05+0.06) +0.0% 0.06(0.09+0.01) +0.0% 0.06(0.08+0.03) +0.0% 2000.9: git add -A (sparse-v4) 0.05(0.05+0.04) 0.05(0.05+0.07) +0.0% 0.05(0.04+0.06) +0.0% 0.06(0.06+0.05) +20.0% 2000.10: git add . (full-v3) 0.38(0.31+0.05) 0.37(0.29+0.06) -2.6% 0.37(0.30+0.07) -2.6% 0.37(0.29+0.06) -2.6% 2000.11: git add . (full-v4) 0.35(0.31+0.04) 0.35(0.29+0.07) +0.0% 0.35(0.29+0.05) +0.0% 0.34(0.29+0.06) -2.9% 2000.12: git add . (sparse-v3) 0.06(0.06+0.05) 0.06(0.05+0.06) +0.0% 0.06(0.07+0.05) +0.0% 0.06(0.09+0.03) +0.0% 2000.13: git add . (sparse-v4) 0.06(0.06+0.06) 0.06(0.07+0.04) +0.0% 0.05(0.06+0.05) -16.7% 0.05(0.05+0.07) -16.7% 2000.14: git commit -a -m A (full-v3) 0.48(0.37+0.08) 0.45(0.36+0.08) -6.2% 0.45(0.35+0.09) -6.2% 0.44(0.36+0.07) -8.3% 2000.15: git commit -a -m A (full-v4) 0.45(0.40+0.06) 0.43(0.34+0.07) -4.4% 0.45(0.37+0.06) +0.0% 0.42(0.36+0.05) -6.7% 2000.16: git commit -a -m A (sparse-v3) 0.05(0.05+0.06) 0.05(0.05+0.03) +0.0% 0.05(0.06+0.06) +0.0% 0.05(0.04+0.06) +0.0% 2000.17: git commit -a -m A (sparse-v4) 0.05(0.06+0.03) 0.05(0.06+0.04) +0.0% 0.06(0.07+0.05) +20.0% 0.05(0.04+0.06) +0.0% 2000.18: git checkout -f - (full-v3) 0.55(0.43+0.08) 0.54(0.46+0.05) -1.8% 0.55(0.46+0.07) +0.0% 0.54(0.40+0.10) -1.8% 2000.19: git checkout -f - (full-v4) 0.55(0.41+0.09) 0.50(0.40+0.09) -9.1% 0.51(0.46+0.05) -7.3% 0.51(0.44+0.06) -7.3% 2000.20: git checkout -f - (sparse-v3) 0.06(0.09+0.03) 0.06(0.08+0.03) +0.0% 0.06(0.06+0.05) +0.0% 0.07(0.09+0.03) +16.7% 2000.21: git checkout -f - (sparse-v4) 0.06(0.08+0.04) 0.05(0.07+0.05) -16.7% 0.05(0.07+0.04) -16.7% 0.06(0.09+0.03) +0.0% ``` All of the above were already integrated. ``` 2000.22: git reset (full-v3) 0.41(0.32+0.06) 0.40(0.31+0.06) -2.4% 0.41(0.33+0.05) +0.0% 0.42(0.34+0.04) +2.4% 2000.23: git reset (full-v4) 0.37(0.32+0.05) 0.35(0.30+0.05) -5.4% 0.37(0.30+0.05) +0.0% 0.35(0.31+0.03) -5.4% 2000.24: git reset (sparse-v3) 0.68(0.65+0.05) 0.55(0.52+0.04) -19.1% 0.04(0.05+0.04) -94.1% 0.04(0.05+0.04) -94.1% 2000.25: git reset (sparse-v4) 0.70(0.65+0.05) 0.54(0.50+0.06) -22.9% 0.04(0.07+0.01) -94.3% 0.03(0.05+0.05) -95.7% 2000.26: git reset --hard (full-v3) 0.54(0.43+0.07) 0.53(0.43+0.06) -1.9% 0.55(0.46+0.05) +1.9% 0.55(0.44+0.06) +1.9% 2000.27: git reset --hard (full-v4) 0.50(0.45+0.03) 0.50(0.43+0.05) +0.0% 0.49(0.41+0.06) -2.0% 0.50(0.42+0.05) +0.0% 2000.28: git reset --hard (sparse-v3) 0.83(0.76+0.06) 0.68(0.62+0.05) -18.1% 0.07(0.05+0.02) -91.6% 0.07(0.05+0.02) -91.6% 2000.29: git reset --hard (sparse-v4) 0.80(0.75+0.05) 0.69(0.62+0.06) -13.8% 0.07(0.04+0.02) -91.2% 0.07(0.04+0.03) -91.2% ``` As expected, `git reset [--hard]` improves with the sparse index integration, but remains constant across the full index case. ``` 2000.30: git update-index --add --remove (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.01+0.01) +0.0% 2000.31: git update-index --add --remove (full-v4) 0.03(0.02+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.03+0.00) +0.0% 0.03(0.02+0.01) +0.0% 2000.32: git update-index --add --remove (sparse-v3) 0.57(0.54+0.02) 0.43(0.42+0.00) -24.6% 0.44(0.41+0.03) -22.8% 0.44(0.42+0.01) -22.8% 2000.33: git update-index --add --remove (sparse-v4) 0.56(0.52+0.04) 0.43(0.42+0.01) -23.2% 0.44(0.42+0.02) -21.4% 0.42(0.41+0.01) -25.0% ``` These do not change significantly because microsoft#423 is not merged. ``` 2000.34: git diff (full-v3) 0.07(0.05+0.03) 0.06(0.05+0.03) -14.3% 0.07(0.05+0.03) +0.0% 0.06(0.05+0.03) -14.3% 2000.35: git diff (full-v4) 0.06(0.05+0.03) 0.06(0.05+0.02) +0.0% 0.06(0.05+0.02) +0.0% 0.06(0.06+0.02) +0.0% 2000.36: git diff (sparse-v3) 0.25(0.23+0.03) 0.17(0.17+0.02) -32.0% 0.18(0.18+0.02) -28.0% 0.01(0.03+0.03) -96.0% 2000.37: git diff (sparse-v4) 0.25(0.22+0.05) 0.16(0.16+0.01) -36.0% 0.18(0.15+0.04) -28.0% 0.01(0.04+0.02) -96.0% 2000.38: git diff --staged (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.02+0.00) +0.0% 2000.39: git diff --staged (full-v4) 0.04(0.03+0.01) 0.03(0.02+0.01) -25.0% 0.03(0.03+0.00) -25.0% 0.03(0.03+0.00) -25.0% 2000.40: git diff --staged (sparse-v3) 0.21(0.19+0.01) 0.15(0.13+0.01) -28.6% 0.15(0.14+0.01) -28.6% 0.01(0.01+0.00) -95.2% 2000.41: git diff --staged (sparse-v4) 0.22(0.21+0.01) 0.14(0.11+0.03) -36.4% 0.15(0.13+0.02) -31.8% 0.01(0.01+0.00) -95.5% ``` The `git diff` improvements are measurable. ``` 2000.42: git sparse-checkout reapply (full-v3) 0.63(0.54+0.05) 0.56(0.48+0.04) -11.1% 0.57(0.48+0.03) -9.5% 0.59(0.48+0.05) -6.3% 2000.43: git sparse-checkout reapply (full-v4) 0.60(0.54+0.02) 0.51(0.46+0.03) -15.0% 0.54(0.48+0.02) -10.0% 0.50(0.44+0.04) -16.7% 2000.44: git sparse-checkout reapply (sparse-v3) 0.91(0.86+0.05) 0.05(0.05+0.00) -94.5% 0.06(0.05+0.01) -93.4% 0.06(0.06+0.00) -93.4% 2000.45: git sparse-checkout reapply (sparse-v4) 0.92(0.88+0.04) 0.05(0.05+0.00) -94.6% 0.05(0.05+0.01) -94.6% 0.05(0.04+0.01) -94.6% ``` Finally, the `git sparse-checkout` measurements are also present. This test script is particularly valuable when contributing changes upstream. It can be good to start by adding the lines to the performance test in an early commit, then demonstrating the performance change by copying the necessary lines from the output table into your commit message.

Sparse index: integrate with `git update-index`

…tests One thing I forgot when talking about the sparse index is that we have a performance test: `t/perf/p2000-sparse-operations.sh`. This test wasn't helpful for commands like `git merge` that need a particular set of input, but work for more read-only operations. Here is a quick demonstration of how this performance test works so we could have a definitive measure of how your previous updates improved performance. To get these results, I ran the following command in `t/perf`: ``` ./run 4bcd533 f9255a5 f28fc01 b713582 -- p2000-sparse-operations.sh ``` The short-shas correspond to the merge commits for these PRs: * microsoft#410 * microsoft#421 * microsoft#417 * microsoft#419 The test takes a copy of the Git repository and creates several copies within a nested directory heirarchy. ``` Test 4bcd533 f9255a5 f28fc01 b713582 ------------------------------------------------------------------------------------------------------------------------------------------------- 2000.2: git status (full-v3) 0.19(0.15+0.05) 0.19(0.16+0.05) +0.0% 0.20(0.18+0.03) +5.3% 0.19(0.17+0.04) +0.0% 2000.3: git status (full-v4) 0.20(0.18+0.04) 0.19(0.15+0.06) -5.0% 0.21(0.18+0.05) +5.0% 0.18(0.18+0.02) -10.0% 2000.4: git status (sparse-v3) 0.04(0.04+0.04) 0.05(0.07+0.04) +25.0% 0.04(0.04+0.05) +0.0% 0.04(0.06+0.04) +0.0% 2000.5: git status (sparse-v4) 0.04(0.03+0.06) 0.04(0.05+0.05) +0.0% 0.05(0.05+0.04) +25.0% 0.05(0.06+0.04) +25.0% 2000.6: git add -A (full-v3) 0.36(0.29+0.05) 0.38(0.28+0.07) +5.6% 0.36(0.31+0.05) +0.0% 0.37(0.31+0.05) +2.8% 2000.7: git add -A (full-v4) 0.34(0.27+0.06) 0.34(0.29+0.05) +0.0% 0.34(0.29+0.04) +0.0% 0.35(0.28+0.06) +2.9% 2000.8: git add -A (sparse-v3) 0.06(0.07+0.04) 0.06(0.05+0.06) +0.0% 0.06(0.09+0.01) +0.0% 0.06(0.08+0.03) +0.0% 2000.9: git add -A (sparse-v4) 0.05(0.05+0.04) 0.05(0.05+0.07) +0.0% 0.05(0.04+0.06) +0.0% 0.06(0.06+0.05) +20.0% 2000.10: git add . (full-v3) 0.38(0.31+0.05) 0.37(0.29+0.06) -2.6% 0.37(0.30+0.07) -2.6% 0.37(0.29+0.06) -2.6% 2000.11: git add . (full-v4) 0.35(0.31+0.04) 0.35(0.29+0.07) +0.0% 0.35(0.29+0.05) +0.0% 0.34(0.29+0.06) -2.9% 2000.12: git add . (sparse-v3) 0.06(0.06+0.05) 0.06(0.05+0.06) +0.0% 0.06(0.07+0.05) +0.0% 0.06(0.09+0.03) +0.0% 2000.13: git add . (sparse-v4) 0.06(0.06+0.06) 0.06(0.07+0.04) +0.0% 0.05(0.06+0.05) -16.7% 0.05(0.05+0.07) -16.7% 2000.14: git commit -a -m A (full-v3) 0.48(0.37+0.08) 0.45(0.36+0.08) -6.2% 0.45(0.35+0.09) -6.2% 0.44(0.36+0.07) -8.3% 2000.15: git commit -a -m A (full-v4) 0.45(0.40+0.06) 0.43(0.34+0.07) -4.4% 0.45(0.37+0.06) +0.0% 0.42(0.36+0.05) -6.7% 2000.16: git commit -a -m A (sparse-v3) 0.05(0.05+0.06) 0.05(0.05+0.03) +0.0% 0.05(0.06+0.06) +0.0% 0.05(0.04+0.06) +0.0% 2000.17: git commit -a -m A (sparse-v4) 0.05(0.06+0.03) 0.05(0.06+0.04) +0.0% 0.06(0.07+0.05) +20.0% 0.05(0.04+0.06) +0.0% 2000.18: git checkout -f - (full-v3) 0.55(0.43+0.08) 0.54(0.46+0.05) -1.8% 0.55(0.46+0.07) +0.0% 0.54(0.40+0.10) -1.8% 2000.19: git checkout -f - (full-v4) 0.55(0.41+0.09) 0.50(0.40+0.09) -9.1% 0.51(0.46+0.05) -7.3% 0.51(0.44+0.06) -7.3% 2000.20: git checkout -f - (sparse-v3) 0.06(0.09+0.03) 0.06(0.08+0.03) +0.0% 0.06(0.06+0.05) +0.0% 0.07(0.09+0.03) +16.7% 2000.21: git checkout -f - (sparse-v4) 0.06(0.08+0.04) 0.05(0.07+0.05) -16.7% 0.05(0.07+0.04) -16.7% 0.06(0.09+0.03) +0.0% ``` All of the above were already integrated. ``` 2000.22: git reset (full-v3) 0.41(0.32+0.06) 0.40(0.31+0.06) -2.4% 0.41(0.33+0.05) +0.0% 0.42(0.34+0.04) +2.4% 2000.23: git reset (full-v4) 0.37(0.32+0.05) 0.35(0.30+0.05) -5.4% 0.37(0.30+0.05) +0.0% 0.35(0.31+0.03) -5.4% 2000.24: git reset (sparse-v3) 0.68(0.65+0.05) 0.55(0.52+0.04) -19.1% 0.04(0.05+0.04) -94.1% 0.04(0.05+0.04) -94.1% 2000.25: git reset (sparse-v4) 0.70(0.65+0.05) 0.54(0.50+0.06) -22.9% 0.04(0.07+0.01) -94.3% 0.03(0.05+0.05) -95.7% 2000.26: git reset --hard (full-v3) 0.54(0.43+0.07) 0.53(0.43+0.06) -1.9% 0.55(0.46+0.05) +1.9% 0.55(0.44+0.06) +1.9% 2000.27: git reset --hard (full-v4) 0.50(0.45+0.03) 0.50(0.43+0.05) +0.0% 0.49(0.41+0.06) -2.0% 0.50(0.42+0.05) +0.0% 2000.28: git reset --hard (sparse-v3) 0.83(0.76+0.06) 0.68(0.62+0.05) -18.1% 0.07(0.05+0.02) -91.6% 0.07(0.05+0.02) -91.6% 2000.29: git reset --hard (sparse-v4) 0.80(0.75+0.05) 0.69(0.62+0.06) -13.8% 0.07(0.04+0.02) -91.2% 0.07(0.04+0.03) -91.2% ``` As expected, `git reset [--hard]` improves with the sparse index integration, but remains constant across the full index case. ``` 2000.30: git update-index --add --remove (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.01+0.01) +0.0% 2000.31: git update-index --add --remove (full-v4) 0.03(0.02+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.03+0.00) +0.0% 0.03(0.02+0.01) +0.0% 2000.32: git update-index --add --remove (sparse-v3) 0.57(0.54+0.02) 0.43(0.42+0.00) -24.6% 0.44(0.41+0.03) -22.8% 0.44(0.42+0.01) -22.8% 2000.33: git update-index --add --remove (sparse-v4) 0.56(0.52+0.04) 0.43(0.42+0.01) -23.2% 0.44(0.42+0.02) -21.4% 0.42(0.41+0.01) -25.0% ``` These do not change significantly because microsoft#423 is not merged. ``` 2000.34: git diff (full-v3) 0.07(0.05+0.03) 0.06(0.05+0.03) -14.3% 0.07(0.05+0.03) +0.0% 0.06(0.05+0.03) -14.3% 2000.35: git diff (full-v4) 0.06(0.05+0.03) 0.06(0.05+0.02) +0.0% 0.06(0.05+0.02) +0.0% 0.06(0.06+0.02) +0.0% 2000.36: git diff (sparse-v3) 0.25(0.23+0.03) 0.17(0.17+0.02) -32.0% 0.18(0.18+0.02) -28.0% 0.01(0.03+0.03) -96.0% 2000.37: git diff (sparse-v4) 0.25(0.22+0.05) 0.16(0.16+0.01) -36.0% 0.18(0.15+0.04) -28.0% 0.01(0.04+0.02) -96.0% 2000.38: git diff --staged (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.02+0.00) +0.0% 2000.39: git diff --staged (full-v4) 0.04(0.03+0.01) 0.03(0.02+0.01) -25.0% 0.03(0.03+0.00) -25.0% 0.03(0.03+0.00) -25.0% 2000.40: git diff --staged (sparse-v3) 0.21(0.19+0.01) 0.15(0.13+0.01) -28.6% 0.15(0.14+0.01) -28.6% 0.01(0.01+0.00) -95.2% 2000.41: git diff --staged (sparse-v4) 0.22(0.21+0.01) 0.14(0.11+0.03) -36.4% 0.15(0.13+0.02) -31.8% 0.01(0.01+0.00) -95.5% ``` The `git diff` improvements are measurable. ``` 2000.42: git sparse-checkout reapply (full-v3) 0.63(0.54+0.05) 0.56(0.48+0.04) -11.1% 0.57(0.48+0.03) -9.5% 0.59(0.48+0.05) -6.3% 2000.43: git sparse-checkout reapply (full-v4) 0.60(0.54+0.02) 0.51(0.46+0.03) -15.0% 0.54(0.48+0.02) -10.0% 0.50(0.44+0.04) -16.7% 2000.44: git sparse-checkout reapply (sparse-v3) 0.91(0.86+0.05) 0.05(0.05+0.00) -94.5% 0.06(0.05+0.01) -93.4% 0.06(0.06+0.00) -93.4% 2000.45: git sparse-checkout reapply (sparse-v4) 0.92(0.88+0.04) 0.05(0.05+0.00) -94.6% 0.05(0.05+0.01) -94.6% 0.05(0.04+0.01) -94.6% ``` Finally, the `git sparse-checkout` measurements are also present. This test script is particularly valuable when contributing changes upstream. It can be good to start by adding the lines to the performance test in an early commit, then demonstrating the performance change by copying the necessary lines from the output table into your commit message.

Sparse index: integrate with `git update-index`

One thing I forgot when talking about the sparse index is that we have a performance test: `t/perf/p2000-sparse-operations.sh`. This test wasn't helpful for commands like `git merge` that need a particular set of input, but work for more read-only operations. Here is a quick demonstration of how this performance test works so we could have a definitive measure of how your previous updates improved performance. To get these results, I ran the following command in `t/perf`: ``` ./run 4bcd533 f9255a5 f28fc01 b713582 -- p2000-sparse-operations.sh ``` The short-shas correspond to the merge commits for these PRs: * #410 * #421 * #417 * #419 The test takes a copy of the Git repository and creates several copies within a nested directory heirarchy. ``` Test 4bcd533 f9255a5 f28fc01 b713582 ------------------------------------------------------------------------------------------------------------------------------------------------- 2000.2: git status (full-v3) 0.19(0.15+0.05) 0.19(0.16+0.05) +0.0% 0.20(0.18+0.03) +5.3% 0.19(0.17+0.04) +0.0% 2000.3: git status (full-v4) 0.20(0.18+0.04) 0.19(0.15+0.06) -5.0% 0.21(0.18+0.05) +5.0% 0.18(0.18+0.02) -10.0% 2000.4: git status (sparse-v3) 0.04(0.04+0.04) 0.05(0.07+0.04) +25.0% 0.04(0.04+0.05) +0.0% 0.04(0.06+0.04) +0.0% 2000.5: git status (sparse-v4) 0.04(0.03+0.06) 0.04(0.05+0.05) +0.0% 0.05(0.05+0.04) +25.0% 0.05(0.06+0.04) +25.0% 2000.6: git add -A (full-v3) 0.36(0.29+0.05) 0.38(0.28+0.07) +5.6% 0.36(0.31+0.05) +0.0% 0.37(0.31+0.05) +2.8% 2000.7: git add -A (full-v4) 0.34(0.27+0.06) 0.34(0.29+0.05) +0.0% 0.34(0.29+0.04) +0.0% 0.35(0.28+0.06) +2.9% 2000.8: git add -A (sparse-v3) 0.06(0.07+0.04) 0.06(0.05+0.06) +0.0% 0.06(0.09+0.01) +0.0% 0.06(0.08+0.03) +0.0% 2000.9: git add -A (sparse-v4) 0.05(0.05+0.04) 0.05(0.05+0.07) +0.0% 0.05(0.04+0.06) +0.0% 0.06(0.06+0.05) +20.0% 2000.10: git add . (full-v3) 0.38(0.31+0.05) 0.37(0.29+0.06) -2.6% 0.37(0.30+0.07) -2.6% 0.37(0.29+0.06) -2.6% 2000.11: git add . (full-v4) 0.35(0.31+0.04) 0.35(0.29+0.07) +0.0% 0.35(0.29+0.05) +0.0% 0.34(0.29+0.06) -2.9% 2000.12: git add . (sparse-v3) 0.06(0.06+0.05) 0.06(0.05+0.06) +0.0% 0.06(0.07+0.05) +0.0% 0.06(0.09+0.03) +0.0% 2000.13: git add . (sparse-v4) 0.06(0.06+0.06) 0.06(0.07+0.04) +0.0% 0.05(0.06+0.05) -16.7% 0.05(0.05+0.07) -16.7% 2000.14: git commit -a -m A (full-v3) 0.48(0.37+0.08) 0.45(0.36+0.08) -6.2% 0.45(0.35+0.09) -6.2% 0.44(0.36+0.07) -8.3% 2000.15: git commit -a -m A (full-v4) 0.45(0.40+0.06) 0.43(0.34+0.07) -4.4% 0.45(0.37+0.06) +0.0% 0.42(0.36+0.05) -6.7% 2000.16: git commit -a -m A (sparse-v3) 0.05(0.05+0.06) 0.05(0.05+0.03) +0.0% 0.05(0.06+0.06) +0.0% 0.05(0.04+0.06) +0.0% 2000.17: git commit -a -m A (sparse-v4) 0.05(0.06+0.03) 0.05(0.06+0.04) +0.0% 0.06(0.07+0.05) +20.0% 0.05(0.04+0.06) +0.0% 2000.18: git checkout -f - (full-v3) 0.55(0.43+0.08) 0.54(0.46+0.05) -1.8% 0.55(0.46+0.07) +0.0% 0.54(0.40+0.10) -1.8% 2000.19: git checkout -f - (full-v4) 0.55(0.41+0.09) 0.50(0.40+0.09) -9.1% 0.51(0.46+0.05) -7.3% 0.51(0.44+0.06) -7.3% 2000.20: git checkout -f - (sparse-v3) 0.06(0.09+0.03) 0.06(0.08+0.03) +0.0% 0.06(0.06+0.05) +0.0% 0.07(0.09+0.03) +16.7% 2000.21: git checkout -f - (sparse-v4) 0.06(0.08+0.04) 0.05(0.07+0.05) -16.7% 0.05(0.07+0.04) -16.7% 0.06(0.09+0.03) +0.0% ``` All of the above were already integrated. ``` 2000.22: git reset (full-v3) 0.41(0.32+0.06) 0.40(0.31+0.06) -2.4% 0.41(0.33+0.05) +0.0% 0.42(0.34+0.04) +2.4% 2000.23: git reset (full-v4) 0.37(0.32+0.05) 0.35(0.30+0.05) -5.4% 0.37(0.30+0.05) +0.0% 0.35(0.31+0.03) -5.4% 2000.24: git reset (sparse-v3) 0.68(0.65+0.05) 0.55(0.52+0.04) -19.1% 0.04(0.05+0.04) -94.1% 0.04(0.05+0.04) -94.1% 2000.25: git reset (sparse-v4) 0.70(0.65+0.05) 0.54(0.50+0.06) -22.9% 0.04(0.07+0.01) -94.3% 0.03(0.05+0.05) -95.7% 2000.26: git reset --hard (full-v3) 0.54(0.43+0.07) 0.53(0.43+0.06) -1.9% 0.55(0.46+0.05) +1.9% 0.55(0.44+0.06) +1.9% 2000.27: git reset --hard (full-v4) 0.50(0.45+0.03) 0.50(0.43+0.05) +0.0% 0.49(0.41+0.06) -2.0% 0.50(0.42+0.05) +0.0% 2000.28: git reset --hard (sparse-v3) 0.83(0.76+0.06) 0.68(0.62+0.05) -18.1% 0.07(0.05+0.02) -91.6% 0.07(0.05+0.02) -91.6% 2000.29: git reset --hard (sparse-v4) 0.80(0.75+0.05) 0.69(0.62+0.06) -13.8% 0.07(0.04+0.02) -91.2% 0.07(0.04+0.03) -91.2% ``` As expected, `git reset [--hard]` improves with the sparse index integration, but remains constant across the full index case. ``` 2000.30: git update-index --add --remove (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.01+0.01) +0.0% 2000.31: git update-index --add --remove (full-v4) 0.03(0.02+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.03+0.00) +0.0% 0.03(0.02+0.01) +0.0% 2000.32: git update-index --add --remove (sparse-v3) 0.57(0.54+0.02) 0.43(0.42+0.00) -24.6% 0.44(0.41+0.03) -22.8% 0.44(0.42+0.01) -22.8% 2000.33: git update-index --add --remove (sparse-v4) 0.56(0.52+0.04) 0.43(0.42+0.01) -23.2% 0.44(0.42+0.02) -21.4% 0.42(0.41+0.01) -25.0% ``` These do not change significantly because #423 is not merged. ``` 2000.34: git diff (full-v3) 0.07(0.05+0.03) 0.06(0.05+0.03) -14.3% 0.07(0.05+0.03) +0.0% 0.06(0.05+0.03) -14.3% 2000.35: git diff (full-v4) 0.06(0.05+0.03) 0.06(0.05+0.02) +0.0% 0.06(0.05+0.02) +0.0% 0.06(0.06+0.02) +0.0% 2000.36: git diff (sparse-v3) 0.25(0.23+0.03) 0.17(0.17+0.02) -32.0% 0.18(0.18+0.02) -28.0% 0.01(0.03+0.03) -96.0% 2000.37: git diff (sparse-v4) 0.25(0.22+0.05) 0.16(0.16+0.01) -36.0% 0.18(0.15+0.04) -28.0% 0.01(0.04+0.02) -96.0% 2000.38: git diff --staged (full-v3) 0.03(0.01+0.01) 0.03(0.02+0.01) +0.0% 0.03(0.02+0.01) +0.0% 0.03(0.02+0.00) +0.0% 2000.39: git diff --staged (full-v4) 0.04(0.03+0.01) 0.03(0.02+0.01) -25.0% 0.03(0.03+0.00) -25.0% 0.03(0.03+0.00) -25.0% 2000.40: git diff --staged (sparse-v3) 0.21(0.19+0.01) 0.15(0.13+0.01) -28.6% 0.15(0.14+0.01) -28.6% 0.01(0.01+0.00) -95.2% 2000.41: git diff --staged (sparse-v4) 0.22(0.21+0.01) 0.14(0.11+0.03) -36.4% 0.15(0.13+0.02) -31.8% 0.01(0.01+0.00) -95.5% ``` The `git diff` improvements are measurable. ``` 2000.42: git sparse-checkout reapply (full-v3) 0.63(0.54+0.05) 0.56(0.48+0.04) -11.1% 0.57(0.48+0.03) -9.5% 0.59(0.48+0.05) -6.3% 2000.43: git sparse-checkout reapply (full-v4) 0.60(0.54+0.02) 0.51(0.46+0.03) -15.0% 0.54(0.48+0.02) -10.0% 0.50(0.44+0.04) -16.7% 2000.44: git sparse-checkout reapply (sparse-v3) 0.91(0.86+0.05) 0.05(0.05+0.00) -94.5% 0.06(0.05+0.01) -93.4% 0.06(0.06+0.00) -93.4% 2000.45: git sparse-checkout reapply (sparse-v4) 0.92(0.88+0.04) 0.05(0.05+0.00) -94.6% 0.05(0.05+0.01) -94.6% 0.05(0.04+0.01) -94.6% ``` Finally, the `git sparse-checkout` measurements are also present. This test script is particularly valuable when contributing changes upstream. It can be good to start by adding the lines to the performance test in an early commit, then demonstrating the performance change by copying the necessary lines from the output table into your commit message.

vdye requested review from derrickstolee and ldennington September 9, 2021 13:39

derrickstolee reviewed Sep 9, 2021

View reviewed changes

ldennington approved these changes Sep 10, 2021

View reviewed changes

read-cache.c Show resolved Hide resolved

t/t1092-sparse-checkout-compatibility.sh Show resolved Hide resolved

derrickstolee reviewed Sep 13, 2021

View reviewed changes

vdye added 4 commits September 13, 2021 12:11

derrickstolee approved these changes Sep 13, 2021

View reviewed changes

derrickstolee mentioned this pull request Sep 13, 2021

Sparse index: add more performance tests #425

Merged

vdye merged commit de87f1d into microsoft:vfs-2.33.0 Sep 13, 2021

vdye deleted the sparse-index/update-index branch September 13, 2021 19:20

dscho pushed a commit that referenced this pull request Oct 30, 2021

Merge pull request #423 from vdye/sparse-index/update-index

746112c

Sparse index: integrate with `git update-index`

derrickstolee pushed a commit that referenced this pull request Oct 30, 2021

Merge pull request #423 from vdye/sparse-index/update-index

a480de7

Sparse index: integrate with `git update-index`

derrickstolee pushed a commit that referenced this pull request Oct 31, 2021

Merge pull request #423 from vdye/sparse-index/update-index

147ea95

Sparse index: integrate with `git update-index`

derrickstolee pushed a commit that referenced this pull request Nov 4, 2021

Merge pull request #423 from vdye/sparse-index/update-index

03648ec

Sparse index: integrate with `git update-index`

derrickstolee pushed a commit that referenced this pull request Nov 4, 2021

Merge pull request #423 from vdye/sparse-index/update-index

dc2e02f

Sparse index: integrate with `git update-index`

derrickstolee pushed a commit that referenced this pull request Nov 10, 2021

Merge pull request #423 from vdye/sparse-index/update-index

2be1bd5

Sparse index: integrate with `git update-index`

derrickstolee pushed a commit that referenced this pull request Nov 15, 2021

Merge pull request #423 from vdye/sparse-index/update-index

934eac0

Sparse index: integrate with `git update-index`

vdye self-assigned this Dec 3, 2021

ldennington pushed a commit to ldennington/git that referenced this pull request Jan 20, 2022

Merge pull request microsoft#423 from vdye/sparse-index/update-index

ab5018c

Sparse index: integrate with `git update-index`

ldennington pushed a commit to ldennington/git that referenced this pull request Jan 20, 2022

Merge pull request microsoft#423 from vdye/sparse-index/update-index

7cceb75

Sparse index: integrate with `git update-index`

ldennington pushed a commit to ldennington/git that referenced this pull request Jan 20, 2022

Merge pull request microsoft#423 from vdye/sparse-index/update-index

8e4df1b

Sparse index: integrate with `git update-index`

ldennington pushed a commit to ldennington/git that referenced this pull request Jan 20, 2022

Merge pull request microsoft#423 from vdye/sparse-index/update-index

5b32dca

Sparse index: integrate with `git update-index`

ldennington pushed a commit to ldennington/git that referenced this pull request Jan 20, 2022

Merge pull request microsoft#423 from vdye/sparse-index/update-index

eb8d6ef

Sparse index: integrate with `git update-index`

ldennington pushed a commit to ldennington/git that referenced this pull request Jan 25, 2022

Merge pull request microsoft#423 from vdye/sparse-index/update-index

351d1c0

Sparse index: integrate with `git update-index`

ldennington pushed a commit to ldennington/git that referenced this pull request Jan 25, 2022

Merge pull request microsoft#423 from vdye/sparse-index/update-index

45ef8c7

Sparse index: integrate with `git update-index`

dscho pushed a commit that referenced this pull request Feb 1, 2022

Merge pull request #423 from vdye/sparse-index/update-index

f35035d

Sparse index: integrate with `git update-index`

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sparse index: integrate with `git update-index` #423

Sparse index: integrate with `git update-index` #423

vdye commented Sep 9, 2021

derrickstolee left a comment

derrickstolee left a comment

derrickstolee Sep 13, 2021

vdye Sep 13, 2021

derrickstolee left a comment

Sparse index: integrate with git update-index #423

Sparse index: integrate with git update-index #423

Conversation

vdye commented Sep 9, 2021

Changes

derrickstolee left a comment

Choose a reason for hiding this comment

derrickstolee left a comment

Choose a reason for hiding this comment

derrickstolee Sep 13, 2021

Choose a reason for hiding this comment

vdye Sep 13, 2021

Choose a reason for hiding this comment

derrickstolee left a comment

Choose a reason for hiding this comment

Sparse index: integrate with `git update-index` #423

Sparse index: integrate with `git update-index` #423