Suggestion: Use Cython

[encukou]

Cython is a Python-like language that gets compiled to C extensions for Python. It's mainly used for speeding up Python code and wrapping C libraries.

For pygit2, Cython would have these advantages:

• python 3 compatibility
• smaller, cleaner code (much less boilerplate; easier to write long docstrings)
• easier maintenance (much DRYer than Python C-API; automatic refcounting)

And some disadvantages:

• it's a complete rewrite
• extra build-time dependency (source releases could come with the compiled .c file, though)
• the language might not be as familiar as C (but it's not that hard to learn)

Do you think it's worth it?

There's a cython branch in my fork; I got the test suite to pass in Pythons 2.5–3.2.

[jdavid]

Hello,

I don't like much the idea. Here are other, closely related, disadvantages:

• Will make pygit2 vulnerable to Cython bugs
• Loss of control on the generated C code
• Likely not as fast (related to the reason above)

It is impressive how you have rewritten pygit2 in just one day, it speaks well of the virtues of Cython. But I don't think it is the best fit for pygit2.

The Python community is quite large and interest on libgit2 should raise overtime; pygit2 is already relatively well maintained, 13 people have contributed so far with patches, others have done so by reporting issues. I think libgit2 deserves hand-written carefully tuned Python bindings, and that there will be enough developers/testers interested to make it happen.

Best regards,

[esc]

Hi,

i have been thinking about writing cython bindings for libgit2. I think the issue of handwritten vs. Cython is not an either or question, but i think that both bindings (I would call the Cython ones cygit2) should co-exist. This would mean the cython bindings would have to be 100% api compatible to the handwritten ones which AFAICS is currently the case since the Cython bindings use the unit-tests. One could think about introducing a flag allowing the programmer to select an appropriate 'backend' to give the choice to the user.

I think both wrapping approaches have their merits and both can learn from each other.

[jdavid]

I find that proposal (pygit2 and cygit2) interesting, it looks there are more than enough developers interested in one or another, or both (searching a little I found someone else had the same idea, https://github.com/pyos/cygit2).

[encukou]

It shouldn't be too hard to maintain the Cython bindings if the test suite is shared with the C-API pygit2, but it'll necessarily mean each would have less users/testers :(
Anyway, I'll be keeping them around at least until pygit2 supports Python 3.
I've contacted pyos to see what he thinks.

As for speed... my code is about 10% slower on the test suite, and I didn't try to optimize at all. The test suite is not a good benchmark, though – there's a lot of exception raising; I think Cython's detailed tracebacks are better than speed here.
And talking about speed is useless without a good benchmark, so I'll end here.

[jdavid]

How to share the unit tests? Maybe you could use Git submodules?

I have open an issue for Python 3 support.

Regarding speed, for what I need pygit2 for, 1% slower would already be too much. It depends on the usage scenario I guess, in my case I use Git as a brick in a database system used in a web application (many requests from clients, blah blah ...).

[encukou]

Git submodules are quite a hassle in my opinion (unless the linked project is big), not to mention libgit2 doesn't really have them yet. I can just continue to pull from pygit2, and ignore the non-test changes.

[ghost]

Actually, I decided to write cygit2 because I like Python 3, but I don't know C good enough. It would be awesome if you could add support for py3k without using cython.

And if you care about speed that much, you chose the wrong language to write your web application in :-)

[jdavid]

Support for Python 3 has been started.

[jdavid]

Just to comment about the speed stuff and more precisely how I see pygit2.

I think pygit2 should be low-level, expose the features of libgit2 to the Python developer and that's it.

Then I guess other developers will use it to write higher level libraries with richer functionality. I believe the developer of GitPython has already started to use pygit2, maybe Dulwich will follow. I myself maintain such a library (itools.git).

This is important. It is related for instance to question discussed in issue #43, whether pygit2 should accept unicode strings for the Hex SHAs or not.

Cheers,

PS : @pyos you are right Python is not the best choice regarding speed, but...

[tmc]

For what it's worth in my initial tests I see very small to no difference in overhead, I haven't taken the time to construct an accurate rig but it should be the case that the same speeds can be accomplished with about half of the code to maintain.

[jdavid]

Time to close this issue. If somebody is interested to develop cygit2 just go ahead. Thanks.