Pyosmium and related dependencies

[dead10ck]

Hi, I wanted pyosmium, so I made a package for it, as well as all of its missing dependencies. This was more involved than my previous PRs, so feedback is welcome.

I wasn't sure if some of these might be better in the regular termux-packages channel, feel free to suggest moving if appropriate.

There were a couple of python packages that I noticed were compiling bytecode and it was ending up in the deb. I tried removing it with TERMUX_PKG_RM_AFTER_INSTALL, but it did not work for some reason I wasn't able to figure out, so I removed them in the make step instead.

[Grimler91]

Hi, do you get any errors if installing pyosmium and pybind11 through pip (after compiling/installing these library dependencies)?

We generally do not package python packages, as python basically does not support cross-compilation, unless they require termux specific patches.

Regarding the error the build of libosmpbf failed with now, you will need termux_setup_protobuf, as run for example here.

The build of the python packages will also fail right now, it will use the python headers in /usr/include instead of $PREFIX/include. See for example the electrum build for how the python setup.py install step is run: https://github.com/termux/unstable-packages/blob/master/packages/electrum/build.sh#L59.

Right now the python dependencies are also packaged into pyosmium I suppose. Would be better to remove them from the package and then install them in a postinst script, as done for electrum.

So, all-in-all I would prefer to not package the python packages, unless there are termux specific issues when installing them through pip.

[dead10ck]

Hi, do you get any errors if installing pyosmium and pybind11 through pip (after compiling/installing these library dependencies)?

We generally do not package python packages, as python basically does not support cross-compilation, unless they require termux specific patches.

It would not work, because pybind11 needs a patch. The cmake file specifies that the python libs only get linked on Windows, and for some reason it seems to be necessary to link it in termux. Without the patch, anything that links against pybind11 will throw errors at runtime about missing symbols.

I'm curious why the stance against all python packages? I haven't tried this, but looking at the electrum example you posted, if it's possible to pass cflags to setup.py, then can't you also pass all the flags needed to cross compile? I'm not sure what most Linux distros do, but surely this is a solvable problem?

After a quick search, this showed up. Not sure if it would be helpful: https://pypi.org/project/crossenv/

And besides this, surely there are tons of pure python modules for which cross compiling is a non issue?

Regarding the error the build of libosmpbf failed with now, you will need termux_setup_protobuf, as run for example here.

Thanks for the tip, I added that.

The build of the python packages will also fail right now, it will use the python headers in /usr/include instead of $PREFIX/include. See for example the electrum build for how the python setup.py install step is run: https://github.com/termux/unstable-packages/blob/master/packages/electrum/build.sh#L59.

Thanks, I added that. But now pybind11 seems to be

Right now the python dependencies are also packaged into pyosmium I suppose. Would be better to remove them from the package and then install them in a postinst script, as done for electrum.

There are actually no other python dependencies in pybind11 or pyosmium, so this shouldn't be necessary.

[dead10ck]

pybind11 seems to be hitting an error on i686 because the python is 64 bit.

[dead10ck]

Another thing to consider is that a couple of CI platforms support building on other architectures, free for open source projects.

• Drone.io has arm64, arm32, and amd64
• Travis has amd64, arm64, and a few others, but no 32-bit archs

Not sure whether this is even an option given the lack of i686, but building on native archs I'm sure could simplify things, and make it possible to run tests on the native architectures.

[Grimler91]

I'm curious why the stance against all python packages?

We don't really want to add packages that are available through other package managers (pip, cpan, tlmgr, gem, ..) to keep the maintenance workload reasonable. Right now we have ~1000 packages maintained by only a handful of people. If we would start adding python (and ruby, and perl, and latex, and node, and ..) packages that can already be installed in another way then that workload would increase with little benefit. (see also package-request notes)

I'm not sure what most Linux distros do, but surely this is a solvable problem?

None of the popular GNU/Linux distros cross-compiles their packages, so they don't run into cross-compilation problems.

crossenv looks interesting, maybe it is possible to create a general termux_step_setup_python function that can be used for python packages with it. I still don't think we should allow packages that can trivially be installed through other package managers though. If pybind11 needs a patch then it can be added though!

There are actually no other python dependencies in pybind11 or pyosmium, so this shouldn't be necessary.

pybind11 seem to have some dependencies (?)

pybind11 seems to be hitting an error on i686 because the python is 64 bit

I would guess that it is finding the python library in /usr/lib/ instead of $PREFIX/lib/, might be solvable by setting some env var like PYTHON_LIBRARY and PYTHON_INCLUDE_DIR, or by patching FindPythonLibsNew.cmake

Another thing to consider is that a couple of CI platforms support building on other architectures

Setting up 4 containers that compiles for the same arch as the host could be one way to simplify building packages that lack cross-compilation support, sure. Might make it harder to tests build locally (from a normal computer) though

[Grimler91]

Looks like pyosmium needs termux_setup_cmake in pre configure step so that it can find cmake

[dead10ck]

We don't really want to add packages that are available through other package managers (pip, cpan, tlmgr, gem, ..) to keep the maintenance workload reasonable. Right now we have ~1000 packages maintained by only a handful of people.

I see, that makes sense.

I'm not sure what most Linux distros do, but surely this is a solvable problem?

None of the popular GNU/Linux distros cross-compiles their packages, so they don't run into cross-compilation problems.

crossenv looks interesting, maybe it is possible to create a general termux_step_setup_python function that can be used for python packages with it. I still don't think we should allow packages that can trivially be installed through other package managers though. If pybind11 needs a patch then it can be added though!

On this note, I tried to see if a pip install worked when my pybind11 package was installed to see if that's all that's required, and unfortunately it did not work that way either. After some digging, it turns out that the pyosmium setup.py doesn't even attempt to use an already installed module, it just downloads it into a local vendor folder and compiles it before compiling its own module. I'm just experimenting now with trying to set cmake options to use the system's module.

pybind11 seem to have some dependencies (?)

These look like dependencies for building the documentation, not the module itself.

pybind11 seems to be hitting an error on i686 because the python is 64 bit

I would guess that it is finding the python library in /usr/lib/ instead of $PREFIX/lib/, might be solvable by setting some env var like PYTHON_LIBRARY and PYTHON_INCLUDE_DIR, or by patching FindPythonLibsNew.cmake

Thanks for the hint, I'm just trying to fix this now.

Setting up 4 containers that compiles for the same arch as the host could be one way to simplify building packages that lack cross-compilation support, sure. Might make it harder to tests build locally (from a normal computer) though

Well, how do most people active in this project typically test builds now? My guess is most people don't have 4 phones, just their one personal device. If this is the case, then the situation right now isn't much different. For a build with an arch you already have, you can install the deb on your personal device and test it manually that way, and for the other archs, there's basically no testing, you just trust that if the binaries built and looks like they're in the right place in the .deb file, then it will work downstream.

I'd think that having 4 hosts on different architectures, it could be a net gain for the QA posture because it would allow you to run the package's automated tests on the same arch that it will be running on downstream. And it could also potentially reduce maintenance burden by simplifying the hoops you have to jump through to get packages to compile cross-architecture. Much of the scaffolding code in the shell scripts could go away. Things like #55 would be a non issue, and much less time would need to be spent figuring out how to patch packages where termux's build infrastructure breaks common assumptions within software packages that you are running on the same arch, you can execute dependent binaries, etc.

It might even be possible to figure out a way to be able to keep testing locally by installing the deb inside a chroot.

[dead10ck]

Okay, I'm kind of at my wit's end about how to get cmake to find the Termux Python. I've tried digging through all the cmake files in the pybind11 repo and setting every variable I could find that looks like it might be for telling it where Python is, but no matter which ones I set to which paths, it always finds the build host's Python. I've tried patching the invocation of find_package(Python ...) to include PATHS and then HINTS, but then it throws an error about missing PythonConfig.cmake.

Unless someone else wants to take a crack at it, I think I'm just going to remove pybind11 and pyosmium from this PR and use my local builds that I got working on my device.

[dead10ck]

It looks like there is some issue with the file server:

curl: (22) The requested URL returned error: 524 
Failed to download https://termux.org/packages/dists/stable/main/binary-x86_64/Packages

[Grimler91]

Well, how do most people active in this project typically test builds now? My guess is most people don't have 4 phones, just their one personal device

Can only speak for myself, but most often I just test new packages on aarch64, and sometimes on arm. So yeah, there's basically no testing..

Note that for native builds we would need containers with a termux environment (so $PREFIX + bionic libc), which xeffyr has created for i686, x86_64 (and arm?).

Okay, I'm kind of at my wit's end about how to get cmake to find the Termux Python

I feel you, cross-compiling for python, ruby and friends are not very fun.

It looks like there is some issue with the file server:

Yeah, happens sometimes with the IPFS host unfortunately