How to use custom backends?

[jnrbsn]

So I see that pygit2 has "some support for custom backends, but undocumented", according to this page, which mainly just links to this pull request. Looking at that pull request, I'm guessing this is how it was intended to work...

Python code excerpt:

from pygit2.repository import BaseRepository
from _my_c_ext import init_my_repo_backend

class MyRepository(BaseRepository):
    def __init__(self, arg1, arg2, arg3):
        backend = init_my_repo_backend(arg1, arg2, arg3)
        super(MyRepository, self).__init__(backend)

Python C extension excerpt:

PyObject *init_my_repo_backend(PyObject *self, PyObject *args)
{
    git_repository *repo;

    /* do stuff to initialize/open the repo ... */

    return PyCapsule_New(repo, "backend", NULL);
}

Is this the intended way this is to be used? It works for me, but I just want to make sure I'm doing it in the intended/supported way and not using stuff that's meant to be "private".

If this looks good, I might contribute some documentation if I have time.

[jdavid]

Looks good. Yes please, PR welcome.

[webknjaz]

Please also note that @hishnash referenced his other usage example at #690 (comment)

[webknjaz]

@jnrbsn I think docs would also benefit of having a complete walk-through of usage of some real DB from https://github.com/libgit2/libgit2-backends. I haven't figured it out myself yet so can't contribute to docs at this point...

[bors-ltd]

Can backends be 100% Python?

(Performance is not the issue I'm trying to solve.)

[webknjaz]

I think they can't because pygit2 is a wrapper around libgit2 that is going to be handling back-ends...

[jnrbsn]

@bors-ltd There's not a built-in way to do that. However, you could write a C extension that implements a libgit2 backend that just calls Python callbacks. Then you could basically have a way of "registering" a 100% Python backend with your C extension. This would still require writing a C extension initially, but then once you have it, you could write as many 100% Python backends as you want. I know you said performance is not the issue you're trying to solve, but I should point out that this solution would not be very performant, especially because I think there's a lot of overhead calling Python from C (although, I'm only vaguely familiar with how that works internally).

[hishnash]

Yep, you would struggle a lot with performance since libgit2 will call there functions A LOT! I think you may well be better off writing a Cython layer as this is close enough to python to be more readable etc for python devs but will interlop with C in a much more performant manner.

…

On 30 Jul 2019, 7:41 PM +0700, Jonathan Robson ***@***.***>, wrote: @bors-ltd There's not a built-in way to do that. However, you could write a C extension that implements a libgit2 backend that just calls Python callbacks. Then you could basically have a way of "registering" a 100% Python backend with your C extension. This would still require writing a C extension initially, but then once you have it, you could write as many 100% Python backends as you want. I know you said performance is not the issue you're trying to solve, but I should point out that this solution would not be very performant, especially because I think there's a lot of overhead calling Python from C (although, I'm only vaguely familiar with how that works internally). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[jdavid]

With PR #942, #948 and #982 merged now it's possible to write custom backends in Python. There's some documentation as well, though not yet released.