-
Notifications
You must be signed in to change notification settings - Fork 34
Conversation
Hi, do you get any errors if installing pyosmium and pybind11 through pip (after compiling/installing these library dependencies)? We generally do not package python packages, as python basically does not support cross-compilation, unless they require termux specific patches. Regarding the error the build of libosmpbf failed with now, you will need termux_setup_protobuf, as run for example here. The build of the python packages will also fail right now, it will use the python headers in /usr/include instead of $PREFIX/include. See for example the electrum build for how the Right now the python dependencies are also packaged into pyosmium I suppose. Would be better to remove them from the package and then install them in a postinst script, as done for electrum. So, all-in-all I would prefer to not package the python packages, unless there are termux specific issues when installing them through pip. |
90156de
to
9427fe0
Compare
It would not work, because pybind11 needs a patch. The cmake file specifies that the python libs only get linked on Windows, and for some reason it seems to be necessary to link it in termux. Without the patch, anything that links against pybind11 will throw errors at runtime about missing symbols. I'm curious why the stance against all python packages? I haven't tried this, but looking at the electrum example you posted, if it's possible to pass cflags to setup.py, then can't you also pass all the flags needed to cross compile? I'm not sure what most Linux distros do, but surely this is a solvable problem? After a quick search, this showed up. Not sure if it would be helpful: https://pypi.org/project/crossenv/ And besides this, surely there are tons of pure python modules for which cross compiling is a non issue?
Thanks for the tip, I added that.
Thanks, I added that. But now pybind11 seems to be
There are actually no other python dependencies in pybind11 or pyosmium, so this shouldn't be necessary. |
pybind11 seems to be hitting an error on i686 because the python is 64 bit. |
Another thing to consider is that a couple of CI platforms support building on other architectures, free for open source projects. Not sure whether this is even an option given the lack of i686, but building on native archs I'm sure could simplify things, and make it possible to run tests on the native architectures. |
ea8378e
to
bc9b567
Compare
We don't really want to add packages that are available through other package managers (pip, cpan, tlmgr, gem, ..) to keep the maintenance workload reasonable. Right now we have ~1000 packages maintained by only a handful of people. If we would start adding python (and ruby, and perl, and latex, and node, and ..) packages that can already be installed in another way then that workload would increase with little benefit. (see also package-request notes)
None of the popular GNU/Linux distros cross-compiles their packages, so they don't run into cross-compilation problems. crossenv looks interesting, maybe it is possible to create a general
pybind11 seem to have some dependencies (?)
I would guess that it is finding the python library in
Setting up 4 containers that compiles for the same arch as the host could be one way to simplify building packages that lack cross-compilation support, sure. Might make it harder to tests build locally (from a normal computer) though |
Looks like pyosmium needs |
a290024
to
70dbdbe
Compare
I see, that makes sense.
On this note, I tried to see if a These look like dependencies for building the documentation, not the module itself.
Thanks for the hint, I'm just trying to fix this now.
Well, how do most people active in this project typically test builds now? My guess is most people don't have 4 phones, just their one personal device. If this is the case, then the situation right now isn't much different. For a build with an arch you already have, you can install the deb on your personal device and test it manually that way, and for the other archs, there's basically no testing, you just trust that if the binaries built and looks like they're in the right place in the .deb file, then it will work downstream. I'd think that having 4 hosts on different architectures, it could be a net gain for the QA posture because it would allow you to run the package's automated tests on the same arch that it will be running on downstream. And it could also potentially reduce maintenance burden by simplifying the hoops you have to jump through to get packages to compile cross-architecture. Much of the scaffolding code in the shell scripts could go away. Things like #55 would be a non issue, and much less time would need to be spent figuring out how to patch packages where termux's build infrastructure breaks common assumptions within software packages that you are running on the same arch, you can execute dependent binaries, etc. It might even be possible to figure out a way to be able to keep testing locally by installing the deb inside a chroot. |
d4ed5af
to
b977449
Compare
Okay, I'm kind of at my wit's end about how to get cmake to find the Termux Python. I've tried digging through all the cmake files in the pybind11 repo and setting every variable I could find that looks like it might be for telling it where Python is, but no matter which ones I set to which paths, it always finds the build host's Python. I've tried patching the invocation of Unless someone else wants to take a crack at it, I think I'm just going to remove |
It looks like there is some issue with the file server:
|
Can only speak for myself, but most often I just test new packages on aarch64, and sometimes on arm. So yeah, there's basically no testing.. Note that for native builds we would need containers with a termux environment (so $PREFIX + bionic libc), which xeffyr has created for i686, x86_64 (and arm?).
I feel you, cross-compiling for python, ruby and friends are not very fun.
Yeah, happens sometimes with the IPFS host unfortunately |
Hi, I wanted pyosmium, so I made a package for it, as well as all of its missing dependencies. This was more involved than my previous PRs, so feedback is welcome.
I wasn't sure if some of these might be better in the regular termux-packages channel, feel free to suggest moving if appropriate.
There were a couple of python packages that I noticed were compiling bytecode and it was ending up in the deb. I tried removing it with
TERMUX_PKG_RM_AFTER_INSTALL
, but it did not work for some reason I wasn't able to figure out, so I removed them in the make step instead.