Question about scmrepo's usage

[aguschin]

Hi guys. Long story short, now we have this GTO tool that allows you to build an Artifact Registry on top of your repo (see example here). I would like to get your advice on how to use scmrepo in it.
Basically, artifacts are listed in artifacts.yaml. To make sense of the repo,

1. GTO needs to traverse all commits (starting from heads and going back) and read the content of this artifacts.yaml in each commit.
2. GTO needs to get a list of git tags will all information (who created that tag, when, etc).

To solve both, I use GitPython now. Everything works pretty well when I have a repo cloned locally, but doesn't work with remote repos. OFC I can clone a repo to a temporary folder and do the same things, but it looks like scmrepo should solve the same kind of tasks for DVC.

So, I would be grateful if you could help me out with this:

1. Am I right and I can use scmrepo for my task? Does it look like the intended usage?
2. What is the right way/right methods/functions to call? Maybe you have some simple examples to start with, that would be awesome.

[pmrowla]

scmrepo is just a wrapper around a few different git backends (dulwich, pygit2 and GitPython) that works at roughly the same level as GitPython. It doesn't really provide any higher level abstraction compared to GitPython, it just provides an API that works consistently regardless of the backend being used.

With git remotes, you can't traverse remote commits without having a local clone (unless you are using something like github's http API, which is completely separate from native git). So your approach (cloning the repo locally) is still what you would need to do with scmrepo, and scmrepo doesn't currently provide any out of the box solution for doing this.

In DVC, we do the same thing as you, and clone repos locally in the dvc.external_repo module (https://github.com/iterative/dvc/blob/main/dvc/external_repo.py). dvc.external_repo provides the abstraction around managing the temporary local git clones (and removing them when we are finished), but the only difference between our implementation and yours would be that we use scmrepo's Git.clone() instead of using the equivalent direct GitPython call.

The main reason scmrepo exists is that GitPython comes with a lot of performance caveats (especially on windows), and in general we try to avoid using GitPython (and prefer dulwich or pygit2 implementations of git calls) whenever possible.

---

It would make sense for us to move the external_repo abstraction for managing these types of temporary git clones from DVC into scmrepo, and we have discussed this as a potential feature before, but right now there's no real plan/timetable on when that might get done.

[aguschin]

Thanks for the detailed explanation, @pmrowla!
I just checked out dvc.external_repo and it doesn't seem to be very complicated, that's nice.
Can I kindly ask to prioritize this within DVC team to be implemented on the next week or two? If yes, that would be great. If not, I'll try to make a PR in scmrepo, use it from GTO and draft PR to use from DVC. Then I'll ask for your help with review.
Would totally do that myself, but I'm afraid I have too many other things with Model Registry right now and time is pressing.
CC @dberenbaum

[dberenbaum]

Hi @aguschin! @pmrowla and I will meet on Monday, so we can discuss then. I'm not sure the level of effort, but my initial reaction is that it's a very tight timeline. Are there any other alternatives? Could GTO depend on DVC temporarily? Maybe you could use dvc.external_repo for now and we can make a plan to extract it before GTO release?

[dberenbaum]

@aguschin We discussed this today, and @pmrowla doesn't think the effort is large, but we can't get to it immediately. We can add it to the dvc backlog and get to it as soon as we have capacity. Does that work for you?

[aguschin]

Could GTO depend on DVC temporarily?
That probably means that we need to instantiate DVC repo and create .dvc folder, right?

Yes, let's add it to backlog then! Thanks @dberenbaum @pmrowla! 🙏

[aguschin]

Hi guys, just checking in on the status. Would be great if you have some plans to address this in upcoming weeks!

[aguschin]

Hi @dberenbaum @pmrowla. Any updates on this?

Would be awesome if this could be completed in 1-2 weeks, cause then @dmpetrov have a conference on 20th of April where this functionality would be very useful. And I'll need some time to add it to GTO after you're done with this.

[francesco086]

Any update here? @pmrowla

I am trying to help @aguschin with this issue on GTO, and git clone cannot be avoided.