dvc: consider switching from GitPython

[Suor]

GitPython causes constant headache and considerable time loss investigating and fixing issues on windows with file not closed and processes not terminated soon enough. We have several places in our code where we were forced to use retries to handle that.

The notable alternatives are pygit2 and Dulwich.

Does anyone know of any downsides of any of these? Or other alternatives? Or maybe there is a good reason we are using GitPython, which I am unaware of?

[shcheklein]

Also, looks like gitpython depends on git being installed:

GitPython needs the git executable to be installed on the system and available in your PATH for most operations. If it is not in your PATH, you can help GitPython find it by setting the GIT_PYTHON_GIT_EXECUTABLE=<path/to/git> environment variable.

Dulwich is pure-python. I wonder if we can do git clone with it as well? It would be great if dvc does not require git at all in certain cases like get/import-url.

We had some troubles with Dulwich trying to compile its C extension or something. @efiop has better idea and even prepared a patch for Dulwich. I'm not sure why we are not just specifying the global option to compile it w/o C-extension.

[efiop]

Dulwich doesn't have wheels yet and to install it in pure-python form, we need to explicitly specify special flag to pip install, which can not be included into install_requires. That is why I've sent a patch that would fallbackt to pure-python if it is unable to complie C extensions, but that patch got declined, because maintainer didn't want people accidentally installing slower dulwich version. So I've helped a bit with automatic wheel building, but there is still stuff to do (e.g. need to fix tests on mac). Other than that, dulwich seems pretty promising.

pygit2 requires libgit2, but it has a pretty good wheel selection already https://pypi.org/project/pygit2/#files , so also might be promising.

We chose to use gitpython because it just worked and is the most popular project among these :) With more advanced use it does has a lot of drawbacks indeed.

I really enjoy using the pygit2 interface.

[Suor]

I am also in favor of pygit2. It's maintained by the same people maintaining libgit2, so it should be future proof and not do anything weird.

[shcheklein]

It looks like it's getting more and more important. A few people mentioned partial checkout problems with gitpython. And it's very important to solve get/dvc API dependency on command line and local filesystem. @efiop can we prioritize the initial research around pygit2?

[efiop]

Sure @shcheklein , done.