How to truncate commit history？

[LittleTigered]

I have a repository with a very long history（20 years, over a hundred million objects）, and it takes several hours to generate the .idx file during a full clone. I'm looking for ways to speed up this process. I have two ideas：

1. Git clone transmits .pack and .idx files simultaneously，allowing the client to bypass the time-consuming git index-pack phase. However, it appears that the Git transfer protocol only supports transmitting the .pack file, can I do this using go-git？

2. Is it possible to discard the commit history before a certain point in time, while keeping the commit IDs unchanged for those after that point (shallow clone is not an option, as I am on the server) BTW I cannot use common optimization methods like git-lfs or rewriting history because the historical versions are already depended upon by other systems and cannot be rewritten.

I know this is not a problem with go-git, any suggestions for solving the issue of Git repository bloat？

[pjbgf]

1. Git clone transmits .pack and .idx files simultaneously，allowing the client to bypass the time-consuming git index-pack phase. However, it appears that the Git transfer protocol only supports transmitting the .pack file, can I do this using go-git

@LittleTigered as far as I am aware, the transfer protocol passes the packfile to the client, which then generates the Index as it parses/scans the packfile. Here's the go-git code that does that:

	w.writer = new(idxfile.Writer)
	var err error

	w.parser = packfile.NewParser(w.synced, packfile.WithScannerObservers(w.writer))

	h, err := w.parser.Parse()
	if err != nil {
		w.result <- err
		return
	}

w.synced is a Reader with the packfile, and the w.writer is the representation of the idxfile.

2. Is it possible to discard the commit history before a certain point in time, while keeping the commit IDs unchanged for those after that point (shallow clone is not an option, as I am on the server) BTW I cannot use common optimization methods like git-lfs or rewriting history because the historical versions are already depended upon by other systems and cannot be rewritten.

I am not completely sure what you are trying to achieve. But it seems like either shallow clones or partial clones are potential options. go-git supports the former, but not the latter. Either way, Github has a great blog post on both.

[LittleTigered]

@pjbgf Thank you for your reply. I will explain my issue more clearly and see if you, as a Git expert, have any good suggestions.

I have a Git server service implemented based on go-git, and the repository has multiple copies on different machines. When a machine goes down and is replaced, I perform a full clone of the repository to generate a new copy, and I want to accelerate this process. As the history grows, the full clone takes several hours, especially as the client spends a significant amount of time generating the index from the received pack.

I think there are two ways to speed up the copying phase:

1. As you described, the client generates the index from the received pack. I wonder if it’s possible to transmit both the pack and the index to the client at the same time, so it saves the time the client takes to compute the index. However, I realize that Git's transfer protocols are all based on packs and do not include the index, so I'm not sure if this approach would present any issues.

2. Truncating history would reduce the size of the repository, making the full clone faster. However, since I am a Git server, it is necessary for the server to maintain the integrity of the repository. Shallow clones and partial clones are optimizations aimed at clients and are not feasible on the server side. Operations like filter-repo rewrite history, which I am not comfortable accepting, because some of the newer versions are already being relied upon by other systems.

I have been looking for a way to truncate history without rewriting the commit IDs, and it’s frustrating that I have yet to find one. For instance, git-replace only gives the illusion that the history is hidden, as the old historical objects cannot be cleaned up by git gc. Is there really no way out, am I just supposed to watch the Git repository grow larger and larger? 🙏🏻