Binary blobs in history

[jsarenik]

While the repository is so young, I would recommend rewriting the history and getting rid of the ~30MB blobs in there.

To find the blobs (and their filenames), save following script as git-find-large.sh:

#!/bin/sh

git rev-list --objects --all |
  git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |
  sed -n 's/^blob //p' |
  sort --numeric-sort --key=2 |
  cut -c 1-12,41- |
  $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest

Now run it and grep for big (MiB) files:

$ sh git-find-large.sh | grep MiB
e18cca20fe54  2.9MiB go-lnmetrics
93d8fe172466  2.9MiB go-lnmetrics
96f8db4416d4  3.1MiB go-lnmetrics
085389b7d797  4.0MiB go-lnmetrics
a08d2baee46a  4.0MiB go-lnmetrics
67fdfa256be6  4.0MiB go-lnmetrics
809698f74644  4.6MiB go-lnmetrics
f42c5dcb17bb  4.6MiB go-lnmetrics
c9260359fd64  4.6MiB go-lnmetrics
ea783e785c3b  4.6MiB go-lnmetrics
8ab6dbfdca16  4.7MiB go-lnmetrics
bb87c77ee906  4.7MiB go-lnmetrics
de3b9e8d1a16  4.7MiB go-lnmetrics
152c6a8bb728  4.8MiB go-lnmetrics
39be3e928398  4.8MiB go-lnmetrics
c6d0b5bd5755  4.8MiB go-lnmetrics
eae5dd9b709b  4.8MiB go-lnmetrics
9a0f78a060b6  4.8MiB go-lnmetrics
d4b142804efe  4.8MiB go-lnmetrics
7e82588f9d75  4.8MiB go-lnmetrics

Now to remove those from history (changing all the following commits), install [git-filter-repo] first and then do this:

$  git-filter-repo --invert-paths --path go-lnmetrics --force

You shall be then left out with rewritten history where your current latest commit c04afd1 becomes 0ad5f11 and the repository ends up to be 333 KiB big (instead of almost 30 MiB).

For a reference, see samhocevar/wincompose#16

[vincenzopalazzo]

Hi @jsarenik,

Thanks for your feedback, I agree maybe before the end of the milestone https://github.com/OpenLNMetrics/go-lnmetrics.reporter/milestone/1 I can clean the history.

Thanks for this amazing feedback!

[vincenzopalazzo]

I follow your step-by-step guide, and I'm happy to have feedback with the new config. Thanks @jsarenik

[jsarenik]

@vincenzopalazzo I was doing a fresh clone to /tmp but did not notice any change. The cloned repository is still ~30 MB big.

[vincenzopalazzo]

Ah, this is bad @jsarenik, I think I need to manage some corner cases in this case with git history, because I change the name of the repository, and back in the history there are some changes in the module.

Do you have some idea on how to remove only the binary from history? maybe we can work together on this if you have time

[jsarenik]

@vincenzopalazzo sure, let's work on it together.