Reduce github repository size

[vfdev-5]

Our .git starts to grow too fast, now. We discussed recently about that with @y0ast and repository size was ~500MB which is rather large. Now, it is 700MB:

/tmp/ignite# du -h -a -d 1
705M    ./.git
4.0K    ./codecov.yml
4.0K    ./.gitignore
28K     ./.circleci
4.0K    ./CODE_OF_CONDUCT.md
4.0K    ./pyproject.toml
916K    ./ignite
4.0K    ./requirements-dev.txt
4.0K    ./CITATION
92K     ./docker
1.2M    ./assets
28K     ./README.md
4.0K    ./LICENSE
12K     ./CONTRIBUTING.md
120K    ./.github
4.0K    ./setup.py
528K    ./docs
1.2M    ./tests
4.0K    ./.pre-commit-config.yaml
15M     ./examples
12K     ./conda.recipe
4.0K    ./setup.cfg
4.0K    ./mypy.ini
723M    .

That time I looked into ressources about reducing the size and it works locally, but it would be helpful to do that on GitHub as well.

Local compressing works:

/tmp/ignite# git gc --aggressive
/tmp/ignite# du -h -a -d 1
112M    ./.git
4.0K    ./codecov.yml
4.0K    ./.gitignore
28K     ./.circleci
4.0K    ./CODE_OF_CONDUCT.md
4.0K    ./pyproject.toml
916K    ./ignite
4.0K    ./requirements-dev.txt
4.0K    ./CITATION
92K     ./docker
1.2M    ./assets
28K     ./README.md
4.0K    ./LICENSE
12K     ./CONTRIBUTING.md
120K    ./.github
4.0K    ./setup.py
528K    ./docs
1.2M    ./tests
4.0K    ./.pre-commit-config.yaml
15M     ./examples
12K     ./conda.recipe
4.0K    ./setup.cfg
4.0K    ./mypy.ini
131M    .

but I have no idea how to apply that to remote...

cc @trsvchn @sdesrozis @fco-dv

[ahmedo42]

This article might be helpful, this will require closing open PR's and branches.

[vfdev-5]

yes, I saw this one and even tried on my fork to reduce the size but without removing objects only sort of applying the result of git gc --aggressive. I dot not remember if that worked for the fork...

[sdesrozis]

@vfdev-5 sorry, I don't have any experience about that... but I really agree to reduce the size of the repo! Maybe @fco-dv would have an idea ?

[sdesrozis]

It seems that git gc is run every day on GitHub server.

[y0ast]

I think I found a smoking gun:

The gh-pages branches: https://github.com/pytorch/ignite/tree/gh-pages
Has many many commits: https://github.com/pytorch/ignite/commits/gh-pages

Containing many small files: 8b7dda3
And 14 individual 3MB files (all `.pickle), like this:
https://github.com/pytorch/ignite/blob/8b7dda3a88ae7d4c4cbd1ea489ec16cc7529208f/master/.doctrees/environment.pickle

That' ~40MB of additions to the .git folder every time the website is deployed (which is for every commit on master!)

So I think a new approach for publishing the website should be considered.

[vfdev-5]

Thanks a lot for the insight @y0ast !
Definitely, previously with Travis we used historyless deployment = 1 commit in the branch. Probably, we shoud find an equivalent with GA.

Maybe, this option to explore: https://github.com/peaceiris/actions-gh-pages#%EF%B8%8F-force-orphan-force_orphan

[sparkingdark]

@vfdev-5 can we first analyze repository with any repo scanning software to get a space insight and From the report when we can try to get the maximum sized folders then we just look into them more thoroughly and try to remove unnecessary packages?

I just freshly clone the project , the size is

Cloning into 'ignite'...
remote: Enumerating objects: 734, done.
remote: Counting objects: 100% (734/734), done.
remote: Compressing objects: 100% (168/168), done.
remote: Total 63540 (delta 600), reused 652 (delta 566), pack-reused 62806
Receiving objects: 100% (63540/63540), 746.72 MiB | 8.55 MiB/s, done.

[vfdev-5]

🎉 I think #1764 has solved the problem 🎉

/tmp# git clone https://github.com/pytorch/ignite.git
Cloning into 'ignite'...
remote: Enumerating objects: 1310, done.
remote: Counting objects: 100% (1310/1310), done.
remote: Compressing objects: 100% (233/233), done.
remote: Total 12057 (delta 1194), reused 1145 (delta 1077), pack-reused 10747
Receiving objects: 100% (12057/12057), 41.17 MiB | 19.61 MiB/s, done.
Resolving deltas: 100% (8434/8434), done.

Thanks again @y0ast for finding the issue ! Very grateful to you !

[y0ast]

95% reduction! Nice fix :)

[trsvchn]

Statistically significant!