Add support for file-based object storage

[JoshuaSjoding]

The go-git library currently relies on in-memory storage of objects. I am interested in adding support for file-based object storage. My long term goal is to use this as a lightweight library for cloning and pulling repositories to local disk.

Right now RAWObjectStorage is the only implementation of the ObjectStorage interface. I would like to contribute by adding a FileObjectStorage implementation that reads and writes objects to the file system.

Is this something that the developers are amenable to?

To meet my goal there would of course be a number of other required features, such as index file support, but those can be tracked as separate issues.

[JoshuaSjoding]

The current implementation of the Repository directly includes RAWObjectStorage. I suggest that the Repository be changed to work through the ObjectStorage interface instead. This way a FileObjectStorage implementation could take advantage of it.

To do this and still support the needs of the Repository, the ObjectStorage interface will need to provide some means of iterating over objects. This means an API change is necessary.

As an experiment last week I familiarized myself with the code by making such a change. You can see the result of the experiment here. You'll notice that I messed with the Iter implementations pretty heavily.

[alcortesm]

Dear Joshua,

We are already working on that, index and everything.

We will contact you again again with more details in a few days, when we
come back from FOSDEM (Brussels) to our offices in Madrid.

[JoshuaSjoding]

Sounds good!

[alcortesm]

Awesome!

I'm eager to take a look to your work!

Thanks for your patience.

[mcuadros]

@JoshuaSjoding is great to see that you are interested on contribute on go-git.

Our first implementation of ObjectStorage was for memory because our main objetive is be able to analyse a git repository very fast, but we are working on now days on a FileObjectStorage and all the required changes. Actually here it the first version of the idx support from some weeks ago.

Anyways any PR related about this or any other topic is more than welcome.

[mcuadros]

About the FileObjectStorage our idea was make the implementation based on .idx files (here is basic implementation of idx reader), something more close to how works git internally when you make a clone than just plain files.

[JoshuaSjoding]

@mcuadros My primary use case is to be able to do the equivalent of a git pull over an existing local repository on disk. That local repository could either be in packed or loose format. In the long term I think that our filesystem.Storage should be able to support both in a single implementation.

Aside from the idx reader, have you guys started the implementation yet or would you like me to go ahead and take a shot at it?

[mcuadros]

@JoshuaSjoding this is perfect, our initial goals was make only base on the idx and pack file from a pulled repository. But make it work with both formats is the final goal.

The only work done was the idx reader (and is a naive implementation), so go ahead, now we are busy with other stuff so will be great if you can handle this task.