Replace _librsynccmodule.c by python code

[ikus060]

I'm creating this task to start a discussion around how to mitigate the "C" extension dependecies in rdiff-backup. The goal is to run rdiff-backup on any platform supported by python. My immediate goal is support ARMv7 architecture.

Here a couple of solutions:

1. Simply replace the _librsync with ctypes layers

• PRO: ctypes is part of python
• CONS: according to various source, ctypes is slower then CFFI

2. Simply replace the _librsync with CFFI layers.

• PRO: Possible to support C extention, but also optional is not available
• CONS: adds a dependencies to rdiff-backup

2. Externalise the librsync library in a separate project (under the rdiff-backup umbrella) Provide a C++ interface and a CFFI interface compatible with each-other

Comments ?

I'm looking to implement the CFFI thin layer.

[ikus060]

Something I did not tought about yet is how to redistribute librscync binaries. For non-unix platform, we may need to include it in the wheel packages.

[ericzolf]

Why should this be a priority? The librsync library can be compiled for ARM and is packaged for (at least) Debian on ARM: https://packages.debian.org/search?keywords=librsync so I guess that it shouldn't be such a challenge to compile the C and _librsync modules for the same platform(s).

[ikus060]

Because I find it very painful to compile rdiff-backup on all the available platform on the market. It adds complexity to the redistribution of the rdiff-backup. Just for Unix like operating system, there are too many flavor I can't count them. Then we have Linux flavors on x86 and ARMv7. Windows 32bits and 64bits.
If we can't get rid of the C extension, that would simplify the redistribution to a single wheel package working for all the platform.

[ericzolf]

The first question if it's really something to do just shortly before releasing 2.0.0, as we've started the beta program. My answer is that it should be done after, and we should focus on the release and getting what we have now into PyPI.

Then, talking about the thing itself:

• for me, sdist is the format used to support "all" platforms, wheel is indeed only meant for a few selected platforms. I would also argue that someone using an obscure Linux version is also able to compile a module (well, pip is doing it anyway for one).
• regarding CFFI I read that it requires a compiler if not using the ABI approach, which doesn't work under Windows. It doesn't sound to me like it would bring us an advantage, but add a dependency.
• nothing to say regarding ctypes, it looks fairly complex and we must make sure that the performance impact isn't too high.

[ericzolf]

Perhaps we could make "pylibrsync" an external library, which would only need to be published and packaged when there is a new version of librsync. This would be the only part which would need to be compiled, once we've got rid of the "cmodule" as well, which sounds more realistic (but other discussion in #67). Perhaps we could even co-maintain these Python-bindings for librsrync with @dbaarda, and profit from his C-experience?

rdiff-backup itself wouldn't any more need any compilation, only pylibrsync.

[dbaarda]

I'd be fine with including/supporting a Python wrapper for librsync in librsync itself.

The packaging etc for this would need to be worked out, but it seems like a reasonable idea to me.

[ikus060]

Link ti #67