Manipulate git-based repositories as FileObjects

[marcenacp]

Context

Croissant supports git-based repositories (e.g., GitHub or Hugging Face). However, we already define a way to extract either from the default branch or from a ref.

Proposal

Reference a specific branch or a tag. We could encode this in the URL just as we encoded the ref. This encoding would be specific to each repository.

• GitHub: https://github.com/<username>/<repository_name>/tree/<branch_name> where branch_name is encoded (e.g., feature/new updates->feature%2Fnew%20updates).
• Hugging Face: https://huggingface.co/datasets/<dataset_id>/tree/<branch_name>.

Reference a specific commit.

• Option 1: It seems that most of git implementations (including GitHub and Hugging Face) still use SHA-1, so we could add cr:sha1 (just like we added cr:md5).
• Option 2: Encode the commit in the URL similarly to the branch or the ref by having: https://github.com/<username>/<repository_name>/commit/<commit>. The inconvenient of this method is that we lose the information about the branch or the ref.

[benjelloun]

I would towards representing the specific branch tag or commit in the URL (or separate dedicated fields), and not just in hash fields like cr:md5, cr:sha1 or sc:sha256. Those fields are intended to provide a checksum to verify content, not as a mechanism to address content, like git does with hashes.

On the exact representation:

• Why do we need to URL encode / for branches or refs?
• If I understand correctly, these are not exactly valid URLs, we decode them before making a git call with those parameters, right?
• Should we consider adding specific fields to represent this information in a more structured way? E.g.: cr:branch, cr:commit, cr:ref, ...