Repo disk size at nearly a gig

[inorganik]

Hey there, I really like this project. I wanted to understand it better so I cloned it down, and was pretty surprised to see the project folder is 722 MB before even running npm install!

I poked around in the folders and pretty much all that size is coming from git history. There must have been some huge files that were committed at one time. I think you can fix this, which probably a good idea when you consider that those huge files aren't part of the project anymore.

Thanks

[saadeghi]

Yes, it's the .git directory and precisely this file which is 700MB:

.git/objects/pack/pack-c787a17add8d4b44731ca2279fa0e5237b91cac1.pack

But I'm not really sure if using a tool like BFG as you mentioned would be safe or not because I haven't use similar tools and I don't want to lose the git history.
Let's say, if there was one or multiple large binary files in git history (which are deleted now) and this tool deletes those file from history. What would happen if I checkout to that commit?
I'm not sure how this tool works and what exact file is getting removed.

[mohe2015]

Yes, it's the .git directory and precisely this file which is 700MB:

.git/objects/pack/pack-c787a17add8d4b44731ca2279fa0e5237b91cac1.pack

But I'm not really sure if using a tool like BFG as you mentioned would be safe or not because I haven't use similar tools and I don't want to lose the git history.
Let's say, if there was one or multiple large binary files in git history (which are deleted now) and this tool deletes those file from history. What would happen if I checkout to that commit?
I'm not sure how this tool works and what exact file is getting removed.

Do NOT use such tools for public repos if its not because of license issues etc as they will break all current clones of that repository. They wont delete history itself but they will rewrite it completely so that for git its a completely different history. If somebody doesn't want to download the full history git has countless features as shallow clones, lazy fetching etc.

[saadeghi]

@mohe2015 good point. If it's rewriting git history, it will be problematic

[inorganik]

I diffed my fork against yours and yeah, many commits and files changed. There's ways around that, but I can see how it would be very disruptive. It's absurd to suggest there are licensing issues though.

[mohe2015]

I diffed my fork against yours and yeah, many commits and files changed. There's ways around that, but I can see how it would be very disruptive.

Which ways are there around that except the --filter I recommended in the other message?

It's absurd to suggest there are licensing issues though.

I didn't suggest that. I said if you would have license issues (copyrighted image or so) you could think about rewriting history but otherwise I wouldnt rewrite history.

[inorganik]

By ways around it, I mean ways around diverged history for those who have PRs open.

[inorganik]

Ok I found a good solution! It is based on this answer. You will need to install git-filter-repo which is a popular git tool that is just a python script, which allows you to rewrite git history.

First you run this command to identify the biggest files tracked by git at one point or another:

git rev-list --objects --all | grep -f <(git verify-pack -v  .git/objects/pack/*.idx| sort -k 3 -n | cut -f 1 -d " " | tail -40)

In the daisyui repo, it output the following:

884db408222685748f8791348987ee85ea86b9bf _app/pages/docs/changelog.svelte.md-840396a2.js
410f70c292a55287f0069cfc92a8666a4751a03e images/stock/photo-1494625927555-6ec4433b1571.jpg
0e8c515d9aae50ca02408e40f092c7592b208ee4 images/stock/photo-1507358522600-9f71e620c44e.jpg
b48397b71a2725493602171773cf71b25297ff8f images/stock/photo-1560717789-0ac7c58ac90a.jpg
10a012a8480c3fb2a2d3686251b9a862dbc64cb8 _app/pages/docs/changelog.svelte.md-64145bc5.js
8433242dda4a5740b16291e73419791be81b6417 _app/pages/docs/changelog.svelte.md-2617064c.js
159cbfe48741a89e4b1e675285ac4eb6fe46328b _app/pages/docs/changelog.svelte.md-f8f019c4.js
87ed1cbbb348ef90b330a01351d7bb8e9b3f3555 _app/pages/docs/changelog.svelte.md-36d84786.js
5622e79cbb8336269bf4ee9ad9274e5221ef7e53 _app/pages/docs/changelog.svelte.md-7b3459dd.js
14623169c0ca1b2335066196097353c836e02693 _app/pages/docs/changelog.svelte.md-75562dc2.js
f43814121ec794c31ebf1f6a6f38c230acf919e4 _app/pages/docs/changelog.svelte.md-9d5e841d.js
de9c9aadd461d0e3ad0aceac7f6d477c69f4d4f5 _app/pages/docs/changelog.svelte.md-0dc365f6.js
68320309aead6d86d71c8e84de0f3c86ae46d2d3 _app/pages/docs/changelog.svelte.md-4d039740.js
2b11b0ea0af4e37e39db2dbcce47a89177e0fbb5 _app/pages/docs/changelog.svelte.md-618e60a7.js
c1c7f2d850feacc944906e48cbec297a8173d4ba _app/pages/docs/changelog.svelte.md-c750fdbf.js
b25a30ed15efcedcec1eec3dbc131efaa58c202e _app/pages/docs/changelog.svelte.md-b66f1f44.js
369a418f91a41ea521c3c2e2c5ea7fce4235cf33 _app/pages/docs/changelog.svelte.md-ccb6c0bd.js
da261b6e81ed8c84a64b4ecf975dfe137e4e0892 _app/pages/docs/changelog.svelte.md-eb0b902b.js
469a88621513ed394c82a9fca602eda0d9842c41 _app/pages/docs/changelog.svelte.md-2f22e7b9.js
0afad811f724c266b18704b42d8d44f2746647a5 _app/pages/docs/changelog.svelte.md-218c2bd0.js
cd9b7e024e59096537ac20a3f4449f6797a21028 _app/pages/docs/changelog.svelte.md-77ec1200.js
1d416f49eb0290947dbab5214945edae8c872507 _app/pages/docs/changelog.svelte.md-60ceb6c1.js
d76134de1d7b31562237208de90107e6d8e7c2d1 _app/pages/docs/changelog.svelte.md-52695023.js
e6494325bbf2420baea09f925f14ee5ac64a4652 _app/pages/docs/changelog.svelte.md-8d888026.js
e1e5e2ca7edbc443b82036638712205415bae7c2 _app/pages/docs/changelog.svelte.md-bd1ff30d.js
dd17b51b850186f661725c3346cebd3328aad474 _app/pages/docs/changelog.svelte.md-1d97d9ea.js
52199e7e70676877cf537a7f69f446eff7505ed0 examples/angular/package-lock.json
5901d78427097e70a89c678b5de862c10569484a nuxt-files/4ebbc47.js
0bc1d1006c688ff9313b1303f34c3a3609de0250 nuxt-files/e079ab9.js
9684b51dbcadf23f861a9369ff0f4d3615b3085b nuxt-files/0bf2d4d.js
3a865747187a951abec29a8e623737676fccd69c nuxt-files/b9ed346.js
5e09a33675b5da938b11f3b1b1dc1d62e8035c4e nuxt-files/cad3594.js
0c9487040eb8eb0d29bd3915b61d2082216b277e nuxt-files/6dbde79.js
f1bc9f6945593a721d66268e7bff91c35d6bee7c nuxt-files/fe401b7.js
8bcb5029edafe0ba159cc604f51b432d6fe2ae32 nuxt-files/d900f1d.js
75517f77b7664a4a88d1700eea747def3b93cf4d nuxt-files/2ea0eaf.js
9700f268bec7c442fbe1473e8b530d26f4584f3e nuxt-files/edcbfee.js
0946910220e1a52111d37c8fc8e68da57be77299 nuxt-files/b9ff7ed.js
00fc15273eec58f7b77a6cf1a8d321b46a3a309e nuxt-files/8bd0ffa.js
296b4e64df027eed57c330b83b02c5e836c62dd1 src/docs/package-lock.json

Let's remove the biggest one, src/docs/package-lock.json:

git filter-repo --invert-paths --path src/docs/package-lock.json --force

Next let's remove the entire "nuxt-files" directory:

git filter-repo --subdirectory-filter 'nuxt-files' --invert-paths --force

To clean up all those svelte changelogs, we can match a path glob:

git filter-repo --path-glob '**/*changelog.svelte.md*' --invert-paths --force

Re-run the rev-list command and you'll see those files are gone. You can keep going on like this and it keeps reducing the size of the .git folder. Seems like there are a lot of files in the history that are no longer tracked.

[mohe2015]

See my comment above. You could do
git clone --filter=blob:none <url>