Support for multiple .dvc roots in a single git repo

[guysmoilov]

This request comes from a large company that has used DVC in the past but moved away mostly due to this issue.

Some companies (.e.g Google) store all of their different projects' and teams' code in a single big git repo.

In this scenario, each project has its own subdirectory in the repo, and they are expected to only make changes to that subdirectory, unless they are contributing code to another project in the organization.

Unfortunately, DVC can only create its .dvc folder in the git repo root. This is problematic for a couple of reasons:

1. The data science team in question might want to work with DVC, but they are blocked from doing so, either just politically or even with automatically enforced authorizations
2. DVC can break backwards compatibility, and different projects in the monorepo might use different versions of DVC

I could probably think of a couple other reasons why this might be a problem.

It seems to me that requiring the .dvc folder be in the root folder is pretty arbitrary, and giving the option to put it in other places in the tree would open the way for wider adoption.

[efiop]

Hi @guysmoilov !

Great suggestion! Are you talking about dvc init specifically or running dvc commands after dvc init --no-scm? The latter one could be ran anywhere, but sequential commands from within that subrepo won't be able to use git. Though we would have to fix both in any case.

[efiop]

Btw, I suppose you are not talking about a repo consisting of git submodules, right? We are able to handle those right now. In that case to support this scenario, we'll have to

1. allow dvc init in directories that are under git control but are not git repo root. We could consider doing this by default, but I'm afraid that some people will do that accidentally and then will be confused as to why they are not able to dvc add something in the parent dir. A safer choice is to introduce something like dvc init --sub.
2. make Git class(and scm in general) search for git repo root up the tree, instead of just trying to use dvc repo root. Enabling this behavior by default might also be confusing, so we might introduce a config option that would be set by dvc init --sub that would tell dvc to behave this way.

Also, for both parts, we would be able to detect misuse and print some nice hints for users.

Overall, seems pretty straightforward. What are your thoughts, guys?

[shcheklein]

@efiop sounds very reasonable to me.

[guysmoilov]

@efiop Hi Ruslan, I was referring to a normal dvc init, in a normal git repo without submodules.
Just adding the ability to treat separate subdirectories in a normal git repo as separate dvc projects.
(And this is just my experience/opinion, but almost no one uses git submodules, it's really awkward)

[efiop]

@guysmoilov Thanks for clarifying! I agree that there are not that many people that using git submodules, but we've got that feature contributed by the user, so that says something 🙂

[sai-prasanna]

@efiop Our team maintains multiple related projects in the same git repository and would love to get this feature running. I would like to get started contributing to this feature. I am new to dvc's code base, will start looking into it this week. Any pointers to what to look for and where would be great.

And what features would this change affect would also be great. e.g. would creating multiple roots in the same repository need GC code to be modified etc.

[pared]

@sai-prasanna thank you for your interest in this issue!

To give you some starting point:

1. You will surely need to comment-out this check
2. when initializing DvcRepo we are assuming that git root is the same as dvc root here. That should be reconsidered.

Some useful methods should be under dvc/scm package, there lies main logic related to code version control.

I hope that will give you some grasp on the issue. Please ping us with any further questions.

[sai-prasanna]

@pared Thanks, will take a look.

Unfortunately I can't start working on it proper immediately. If anyone else want to work on this immediately, feel free.