sparse index support

[Sebastian Thiel (Byron)]

Quick recap: What is a sparse index?

Instead of containing one entry for every file in the worktree ("regular" index structure), a sparse index only contains a subset of these. Additionaly, it contains entries to directories that are marked with the SKIP_WORKTREE flag. All files within these entries can be skipped by functions that read / update the index and thereby increase performance.

If the index file contains the the "Sparse Directory Entries" extension marked by the signature sdir, it is classified as a sparse index.

Motivation

The goal of this issue is to keep track of the requirements necessary to eventually fully integrate sparse index support for gitoxide.

This issue does not yet contain all the tasks and considerations by any means, but the goal is to add new knowledge and keep everything up to date as I go along and things become more clear.

Tasks

• reading
  • "regular" index with files containing the SKIP_WORKTREE flag
  • sparse index with directories containing the SKIP_WORKTREE flag, in cone mode
  • write specific tests to verify those behaviours
• Write sparse index #563
  • Tree extension order in gitoxide is different than in git, prevents raw byte comparisons
  • configure index version via write::Options
• find out what options in git-config influence / configure sparse index related tasks to better understand what capabilities are needed
  • update gix progress with those findings
• git-repository loads worktree configs #635
• implement functionality similar to ensure_full_index()
  • scan index for sparse directory entries (trees) and expand them into a full list of filepaths (regular index structure), mutating the current index State, for use in subsequent functions that don't support working with sparse indexes yet
  • find out where and how it make sense to use that function
• matching logic of .git/info/sparse-checkout for cone mode
  • cone mode
  • no-cone mode (inverted .gitignore) this functionality is deprecated in git
• restore DIR information during writing or as separate step as indicated here
• command similar to `git sparse-checkout set / add
  • support --cone and --no-cone flags

Notes

• the git sparse-checkout set / add commands modify the list of files contained in .git/info/sparse-checkout, which uses the same syntax as a .gitignore file. Cone mode and non-cone mode decide how this file gets interpreted. Cone mode will match only directories while non-cone mode will use the same matching logic used for .gitignore files. read more
• non-cone mode and sparse index are incompatible with eachother
  that makes sense because sparse indexes mark entire directories as SKIP_WORKTREE which is what cone-mode matches on, while non-cone mode can also match on single files which does not give an advantage to the amount of entries in the index
• non-cone mode is now deprecated

References

• docs: git-sparse-index
• docs: git-sparse-checkout
• docs: git-index | sparse entries
• docs: git-config

---

Originally proposed by Sidney Douw (@SidneyDouw) in issue #562.