disk usage when using sparse_paths

[mastupristi]

I'm trying out gitman
I set up GITMAN_CACHE, and my file contains.

location: gitman_modules

sources:
  - name: cmsis_5
    type: git
    repo: https://github.com/ARM-software/CMSIS_5.git
    rev: 2b7495b8535bdcb306dac29b9ded4cfb679d7e5c
#    sparse_paths:
#      - CMSIS/Core/Include
#      -  CMSIS/DSP/Include

  - name: mcux-soc-svd
    type: git
    repo: https://github.com/NXPmicro/mcux-soc-svd.git
    rev: e93e0b0067b9297de056de4cef6425c9c999fda4
#    sparse_paths:
#      - MIMXRT1175
    

  - name: nxp-mcux-sdk
    type: git
    repo: https://github.com/NXPmicro/mcux-sdk.git
    rev: 6f3fd257cdcf978a4d26e7d6e9eed9240037422b
#    sparse_paths:
#      - drivers
#      - devices/MIMXRT1175
#      - components/phy/device/phyrtl8211f
#      - components/phy/device/phyksz8081

I try both with sparse_paths commented and uncommented and these are the results:

Repository	Mode	Working Directory	.git Size	Total Size
cmsis_5	Full	133 MB	1.1 MB	134 MB
cmsis_5	Sparse	3.5 MB	261 MB	265 MB
mcux-soc-svd	Full	2.1 GB	208 KB	2.1 GB
mcux-soc-svd	Sparse	42 MB	37 MB	79 MB
nxp-mcux-sdk	Full	1.2 GB	2.7 MB	1.2 GB
nxp-mcux-sdk	Sparse	98 MB	237 MB	335 MB

Questions

• What is causing the .git directory to grow so much when using sparse_paths?
• Is this a Gitman-specific behavior (e.g., due to how git clone is invoked)?
• Does this come from Git itself, perhaps because of --dissociate when cloning with --reference?
• Are there parameters or options (in Git or Gitman) that can avoid this .git bloat while keeping sparse checkouts?
• Is there a known clean way to preserve the benefits of GITMAN_CACHE without duplicating all objects in .git/objects?

[jacebrowning]

Good questions, but I don't know too much about how sparse checkouts work in the git internals. @xenji any ideas as the original implementer of this feature?

[mastupristi]

Hi @jacebrowning

I've been investigating .git size usage when using sparse checkouts and --reference clones with Gitman.

Observation

I noticed that Gitman always creates the cache with:

git clone --mirror <repo> <cache_path>

Then, behavior diverges depending on whether the module uses sparse_paths:

• If no sparse checkout is used, Gitman clones with:

  git clone --reference-if-able <cache_path> <repo> <target_path>
  

• If sparse checkout is enabled, Gitman does not use --reference-if-able and instead initializes the repo manually, using the cache path as a remote and calling git pull and checkout.

Question

Why does this behavior differ?

Is this intentional, or perhaps a leftover from a previous implementation?

Using --reference-if-able even when performing sparse checkouts seems to work well in my experiments, and it helps prevent the .git directory from growing excessively.

Experiment

I wrote a small script (just rename it to .sh) that mimics Gitman's behavior but always uses --reference-if-able, even for sparse modules.

Results

Repository	Mode	Working Directory	.git Directory	Total Size
mcux-soc-svd	Full	2.1 GB	192 KB	2.1 GB
	Sparse	42 MB	200 KB	42 MB
mcux-sdk	Full	1.2 GB	2.6 MB	1.2 GB
	Sparse	43 MB	2.7 MB	46 MB
CMSIS_5	Full	137 MB	924 KB	138 MB
	Sparse	4.9 MB	948 KB	5.8 MB

This is a significant improvement compared to the current Gitman behavior for sparse checkouts.

I'd love to understand whether this approach is compatible with Gitman's design and whether support for --reference-if-able with sparse modules could be added.

Thanks again for your work on this great tool.

[mastupristi]

Hi @jacebrowning , @xenji

Following up on my previous post, I took a look at the source code of gitman/git.py and found that the behavior regarding sparse checkouts is indeed intentional and not a leftover.

In particular, this block explains it clearly:

    if sparse_paths and sparse_paths[0]:
        os.mkdir(normpath)
        git("-C", normpath, "init")
        git("-C", normpath, "config", "core.sparseCheckout", "true")
        git("-C", normpath, "remote", "add", "-f", "origin", sparse_paths_repo)

        with open(
            "%s/%s/.git/info/sparse-checkout" % (os.getcwd(), normpath),
            "w",
            encoding="utf-8",
        ) as fd:
            fd.write("\n".join(sparse_paths))
        with open(
            "%s/%s/.git/objects/info/alternates" % (os.getcwd(), normpath),
            "w",
            encoding="utf-8",
        ) as fd:
            fd.write("%s/objects" % sparse_paths_repo)

        # We use directly the revision requested here in order to respect,
        # that not all repos have `master` as their default branch
        git("-C", normpath, "pull", "origin", rev)
    elif settings.CACHE_DISABLE:
        git("clone", repo, normpath, *user_params)
    else:
        git("clone", "--reference-if-able", reference, repo, normpath, *user_params)

So the sparse checkout is done by initializing a new repository, manually setting the origin remote, and configuring the mirror as an alternate object store — instead of using git clone --reference-if-able.

This clarifies why, in my tests, sparse checkouts created by Gitman show .git directories significantly larger than those created with --reference-if-able.

---

My question now

Are there any known drawbacks or edge cases that justify this manual setup over a simpler:

git clone --reference-if-able <mirror> <repo-url> <target-dir>
cd <target-dir>
git sparse-checkout init --cone
git sparse-checkout set <paths>...

In my experiments, using --reference-if-able even for sparse checkouts:

• Greatly reduces the .git directory size.
• Avoids manually writing to alternates.
• Seems to work reliably (tested with Git ≥ 2.42).

Is there any technical reason why Gitman avoids using --reference-if-able in the sparse-checkout case?

Would it be possible to use the same strategy (i.e., git clone --reference-if-able, followed by sparse-checkout init/set) even for sparse checkouts?

If not, what are the potential drawbacks of that approach?

However, I'm going to try more experiments, and if I'm successful I would try to propose patches

Thanks again for maintaining this project!

[xenji]

Hi there. Thank you for your thorough investigation. I don't remember much the code I wrote back then to be honest.
There are to possible explanations:

1. One did not know better at the time - and I must admit that the option you presented was still unknown to me until you mentioned it
2. git might not have supported it back then

Either way, I cannot speak of any intention leaving it out.

[mastupristi]

Hi @xenji

I can understand you
My previous script actually eventually produces small directories, but there is a time when the repos are very large.
Maybe you wanted to avoid this

I created a new script to solve this problem as well

In this script I do git fetch origin without specifying anything else, so all the refs are downloaded, and I have no problem with the following checkout (be it a branch, a tag, or a hash), at the cost of an increased size of the .git directory

New script achieve the following results:

Repository	Working Dir	.git Dir	Totale
mcux-soc-svd	42M	276K	42M
mcux-sdk	37M	2.9M	39M
CMSIS_5	4.9M	1.1M	5.9M

The results are not as good as the previous script but we can work on it

I modified the git.py script like this:

    if sparse_paths and sparse_paths[0]:
        os.mkdir(normpath)
        git("-C", normpath, "init")
        git("-C", normpath, "remote", "add", "origin", repo)

        if not settings.CACHE_DISABLE:
            with open(
                "%s/%s/.git/objects/info/alternates" % (os.getcwd(), normpath),
                "w",
                encoding="utf-8",
            ) as fd:
                fd.write("%s/objects" % sparse_paths_repo)

        git("-C", normpath, "sparse-checkout", "init", "--cone")
        git("-C", normpath, "sparse-checkout", "set", *sparse_paths)
        git("-C", normpath, "fetch", "origin")
        git("-C", normpath, "checkout", rev)
    elif settings.CACHE_DISABLE:
        git("clone", repo, normpath, *user_params)
    else:
        git("clone", "--reference-if-able", reference, repo, normpath, *user_params)

I have only tried doing an install and it seems to work. Then to say it`s okay and reliable there will be more testing to be done

I envision gitman used primarily in two contexts:

• development, where dependencies are convenient to have all the refs, to be able to navigate between branches, tags, etc.
• CI, where instead the specific commit is taken and I don't care about either history or other branches

In order to further reduce bandwidth and disk usage, I can imagine adding one or two parameters to trigger git parameters such as --depth=1 or other

What do you think of this snippet? If you think it is good, I will proceed to make a pull request

best regards
Max

[xenji]

I don't know much about the other use cases besides CI. For me, this has always been the primary one. @jacebrowning Maybe you can elaborate here?

I agree to the CI-related parts. Also, the depth=1 addition would be very valuable. Please keep in mind that some CI systems use a single checkout to refer to and reuse. This would render the created checkout unusable for this purpose. Not a blocker, but it needs proper documentation.
If the CI process runs fully self-contained, including the checkout, the optimization of space and time usage would be significant.

[jacebrowning]

I haven't used the sparse checkouts feature myself, but I'd be open to a PR with this improved clone behavior for it.

[mastupristi]

@jacebrowning PR #354 opened