feat(pkg): Implement downloading sources via git

[Leonidas-from-XIV]

This adds an option to retrieve git sources via the rev store by adding the commits to the rev store and then materializing a repo in the target directory.

[Leonidas-from-XIV]

clone should be fairly cheap on the same file system as it should hardlink objects and it prevents the checked out source from interacting with the rev store. The other option is to create a git worktree, however it retains a connection to the rev store and this is something we probably want to avoid.

The other option is to just checkout the files without any .git repo attached to it, but then it doesn't quite match the semantics of git clone <url>.

[rgrinberg]

The other option is to just checkout the files without any .git repo attached to it, but then it doesn't quite match the semantics of git clone

I think this is the way to go. Note that if you add the .git/ stuff to the package source, you will make the build rule depend on everything inside it (which is junk in this case). Thereby adding a crapload of unnecessary dependencies to the build rules.

I suggest unpacking the git repo using git archive into the correct location.

[Leonidas-from-XIV]

Yes, that's a good point. The other issue is that it presumably clones everything, not just the repo but all other revisions as well. You can create a shallow clone but apparently only of a branch.

The downside of git archive is that it pulls in tar as a dependency, but at least it doesn't waste time compressing and the options to decompress should work on any POSIX tar. I've updated the branch.

[rgrinberg]

Even a shallow clone wouldn't be enough as we don't really want to depend on the raw git objects. Those can contain a lot irrelevant data for any non trivial repo.

Depending on tar is unfortunate, but we already do it for fetching via curl. We should definitely get rid of the extra dependencies eventually.

[Leonidas-from-XIV]

Also cloning (shallow or not) only works when supplied with refs. Internally clone calls uses fetch and looking at the definition what to fetch, the refspec it says:

Specifies which refs to fetch and which local refs to update. When no s appear on the command line, the refs to fetch are read from remote..fetch variables instead (see CONFIGURED REMOTE-TRACKING BRANCHES below).

So if we wanted to fetch specific revisions (at least with tie git CLI), we would probably need to create a tag for every single commit in the rev store.

As such I would suggest merging this PR with git archive+tar for now and we can revisit this functionality later should there be need for optimization.