eCAL Packages on PyPi

[KerstinKeller]

Context

We want to be able to do

pip install ecal

for a seamless python integration.
However the wheels that we produce are not PyPi ready.

Proposal

We need to make a few changes to the build in order to make it pypi ready:

1. Drive the build from setup.py. Atm, CMake generates setup.py and then calls it. We need to reverse this process. We need setup.py that calls cmake. Only that way can we make source distributions of eCAL.
2. Use either https://scikit-build-core.readthedocs.io/en/latest/index.html or https://github.com/tttapa/py-build-cmake to drive the build.
   We will need hdf5 and protobuf submodules, and need to check the options for a proper build.
3. Setup build jobs that build releases (and then push to pypi).
   For linux build, build inside a manylinux docker image:
   https://github.com/pypa/manylinux
4. Build all wheels in github actions using cibuildwheel

Tasks and updates

• use py-build-cmake
• py-build-cmake creates first a source distribution and then tries to build that
• currently _pb2 file generation had to be removed, they are no more installed to the package.
  Do we want them? It's certainly doable but might be a bit more work
• version from git tag (or from whereever) - right now a version is noted in init.py -> need to check options in setup.cfg
• Find a solution for the pre-generated pb2.py files from the python samples (reference to Support newer Protobuf #1187)

[FlorianReimold]

For windows x64 we should be PyPi ready, as Windows has the most amazing binary compatibility of all Operating systems. So there really isn't an issue with publishing .whl files containing native .dlls

For Linux we currently depend on too many system packages, I guess. The .whl files are just not self contained enough for being functional on a wide variety of Linux distros.

[DownerCase]

You've probably already done the investigation; but it is certainly possible to generate the pb files as part of setup.py, I've been able to do a PoC of that process. Mostly involves injecting an extra subcommand into the main build command.

You can look to Google's protobuf distutils and the newer (and pyproject compatible) setuptools-protobuf projects for inspiration (if they're not directly suitable).

[KerstinKeller]

@DownerCase thanks for this great hint! Setuptools-protobuf seems really promising.
We have to see if it plays together with py-build-cmake, but worst case, we'll make two different packages, one with the binary extensions, and one with the _pb2 files, but let's see..

[DownerCase]

That said I was commenting from a pure python standpoint.

Given you’re calling into a CMake build system you should be able to set up an add_custom_command to call protoc with the normal —python-out argument to generate the python bindings as part of the build.

[KerstinKeller]

The good thing is that during build, we already have protobuf available. That's challenging with setuptools-protobuf as the user has to provide a protoc (which needs to be compatible with the protobuf version that he wants to use).