Writing a module of type Git.S

[njayinthehouse]

Description
I'm interested in implementing a module for performing git-like operations in Cassandra. Specifically, I'm writing a module that can be passed to functor Irmin_git.Content_addressable, in order to maintain a content-addressable store on Cassandra.

Specific questions

1. What exactly is a buffer, and how are we using it?
2. What is a PACK stream/What does the Pack module do?

[dinosaure]

Hi!

So if I understand correctly, you want to simulate Git with Cassandra? I'm not really aware about Cassandra but I can give you some details about Git.

What exactly is a buffer?

Currently, buffer is a set of Cstruct.t used to make a Git.t - this type is abstracted and can be constructed only on a specialized target (eg. git with a file-system or git-mem).

It help to bound allocation needed by common Git algorithms. By this way, read or write operations will use this set of buffers instead to allocate in-the-fly Cstruct.t - and, by this way, try to fix an old issue Out_of_memory.

What is a PACK stream/What does the Pack module do?

In Git, we have 2 ways to store Git objects:

• loose objects
• pack-file

The first is used when you make a git commit, some loose files appears in your disk. The second appears when you git clone/git fetch. The PACK stream is the format exchanged between 2 Git entities (like you and GitHub).

An other case is when you do a git gc when all of your loose files will be packed into a new PACK file - currently, even if it's possible to make a PACK file, the git gc is not yet implemented.

Finally, from a PACK file, we can extract several Git objects (commit, tag, tree or blob). According to what I said, the PACK stream is mostly used by the Sync.mli module to be able to download/upload Git objects.

About your goal

Required types and functions are quite independent from the Git format where you can consider type buffer = unit if you want and a PACK file as a simple format (like a succession of Git objects without compression and delta-ification).

However, because irmin wants to have a way to synchronize two peers over a protocol, you must implement the PACK stream.

[njayinthehouse]

Thanks for your detailed response, I now have a much clearer idea of how to go about this.