split scm from dvc

[skshetry]

We want to split scm from dvc into a separate package on its own. Ideally, we'd like to have a proper and stable API. Though I'd prefer we split as-is.

I want to split the scm from dvc, using filter-repo to keep history intact.

Though I still see a few issues that we need to fix:

• DVC uses logging info for UI purposes, which dvc.scm still has.
  So we may need to get rid of logging.info calls, and make scm work in terms of proper exceptions/returns. (Important)
• Migrate exceptions as they subclass DvcException and are part of our UI framework.
• scm uses Tqdm progress bars for clone operation. We need to update it to use callbacks. (Medium priority)
• Migrating tests and fixtures used. We need to copy it or figure out a way to share fixtures.

As I want to split those from DVC with history intact, we should fix 1 and 2 in DVC itself and then need to split it out.

filter-repo how-to

I used the following snippet to split scm before.

cd "$(mktemp -d)"
git clone git@github.com:iterative/dvc.git .
git filter-repo --path dvc/scm --path dvc/tree/git.py --path dvc/fs/git.py --tag-rename '':'scmrepo-' --path-rename dvc:scmrepo

The following structure is generated:

tree .
.
├── scmrepo
│   ├── fs
│   │   └── git.py
│   └── scm
│       ├── __init__.py
│       ├── base.py
│       └── git
│           ├── __init__.py
│           ├── backend
│           │   ├── __init__.py
│           │   ├── base.py
│           │   ├── dulwich.py
│           │   ├── gitpython.py
│           │   └── pygit2.py
│           ├── objects.py
│           └── stash.py

[efiop]

Some background on the split: our scm module supports multiple backends which are used in GitFileSystem. That functionality is fairly general (could be used elsewhere) and stable, but is expensive to test.

[skshetry]

We depend on more than GitFileSystem, specifically Git which supports multiple backends, which is also general.

[skshetry]

After #6961, we won't have any logger.info calls in scm, we still have some logger.debug calls. Most of those are in scm.stash and is being used by Experiments. I could remove those logs or move those calls to where it's being used.

But looking at the Experiments, I wonder if we should split the stash-ing part out of it to a separate place? This is not required for this, but it is nice to have. @pmrowla, WDYT? Does extracting that out make sense? Or, do you see another way we could do this?

[pmrowla]

@skshetry I think for now we can just remove the debug logging calls from the separated SCM module.

Separating the experiments code base into parts for "managed git branches" (including the stash related behavior) and "pipeline/exp execution" is nice to have in the future, but I don't think it needs to be done right now.