Proposal to improve memory performance decoding packfiles.

[ajnavarro]

Proposal

Problem

Analyzing the code, I saw the following on packfile decoder:

• If no EncodedObjectStorer is provided, a new memory object is created: https://github.com/src-d/go-git/blob/master/plumbing/format/packfile/decoder.go#L324
• If an EncodedObjectStorer is provided (the most common case is Filesystem EncodedObjectStorer implementation), we are doing the same: https://github.com/src-d/go-git/blob/master/storage/filesystem/object.go#L76

This is forcing the read of all the object contents, even if we are not gonna use them.

Proposed solution

Instead of copy all into memory and resolve all the object deltas, create a DeltaReader abstraction to encapsulate that logic:

All the actual logic of the Object Cache and delta resolution will be exposed as a Reader. This Reader will contain the same logic as the actual one that is in the decoder, but it will be only executed if the Object.Read() method is called.

This reader will use the ObjectStorer to resolve deltas. The deltaBaseCache that is on the decoder should be moved inside the ObjectStorer.

Applying this solution will decrease drastically the memory usage. Instead of having all the objects in memory + deltaBaseCache objects + delta base + delta target, we are going to have only deltaBaseCache objects + delta base and target.

Caveats @jfontan @smola @mcuadros ?

[mcuadros]

Solved by:
plumbing/format: packfile, new packfile representation #899 (erizocosmico)
plumbing/format: packfile, new packfile parser #898 (jfontan)
plumbing/format: packfile, improve packfile reading performance #906 (erizocosmico) #916 (jfontan)