Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Help needed with building an spk for home assistant (python3-based app) #1654

Closed
hansmbakker opened this issue Apr 17, 2015 · 19 comments
Closed

Comments

@hansmbakker
Copy link

I'm trying to build an spk for Home Assistant / https://github.com/balloob/home-assistant.git. It is a promising home automation server written in Python 3.

I've created a basic setup for it at f013a38.

The dependencies build now, but it starts the "stripping binaries" build step based on the PLIST from the work directory instead of from the spk/homeassistant directory. This causes spksrc trying to find binaries that are not built.

Can anybody give me some hints in the right direction?

@Dr-Bean
Copy link
Contributor

Dr-Bean commented Apr 18, 2015

Your description isn't completely clear to me, but I believe you might be referring to the difference between DEPENDS and BUILD_DEPENDS.

As an example, the Mercurial package uses that approach:

BUILD_DEPENDS = cross/python cross/setuptools cross/pip cross/wheel cross/$(SPK_NAME)
.

Fwiw, try to replicate the approach of cross/mercurial with your netifaces module: Instead of adding cross/netifaces as a module in the Python SPK, make that a (cross-compiled) wheel, and add the resulting wheel to your SPK in the same way as cross/mercurial is added to spk/mercurial.

@hansmbakker
Copy link
Author

@Dr-Bean you're right, I confused DEPENDS and BUILD_DEPENDS. This helped a lot, thanks!

After changing that, I could build the package (including PyYaml and netifaces as cross-compiled wheels), but adding pyephem as a wheel didn't go well.

I took my netifaces wheel as an example, but I think there is something wrong in the pyephem Makefile that causes spksrc not to understand what it should do. It doesn't even download the package source..

The changes I made are at master...wind-rider:homeassistant

It doesn't matter whether I try to quickly build pyephem from its cross folder or by referencing it from my spk folder.

Could you explain to me what's going wrong?

See the log below:

root@89bc6447155d:/spksrc/cross/pyephem# make arch-armadaxp-5.1
../../mk/spksrc.depend.mk:56: warning: overriding recipe for target '/spksrc/cross/pyephem/work/.pyephem'
../../mk/spksrc.download.mk:147: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work/.pyephem'
../../mk/spksrc.checksum.mk:84: warning: overriding recipe for target '/spksrc/cross/pyephem/work/.pyephem'
../../mk/spksrc.depend.mk:56: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work/.pyephem'
../../mk/spksrc.extract.mk:58: warning: overriding recipe for target '/spksrc/cross/pyephem/work/.pyephem'
../../mk/spksrc.checksum.mk:84: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work/.pyephem'
../../mk/spksrc.patch.mk:62: warning: overriding recipe for target '/spksrc/cross/pyephem/work/.pyephem'
../../mk/spksrc.extract.mk:58: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work/.pyephem'
../../mk/spksrc.configure.mk:58: warning: overriding recipe for target '/spksrc/cross/pyephem/work/.pyephem'
../../mk/spksrc.patch.mk:62: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work/.pyephem'
../../mk/spksrc.compile.mk:44: warning: overriding recipe for target '/spksrc/cross/pyephem/work/.pyephem'
../../mk/spksrc.configure.mk:58: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work/.pyephem'
../../mk/spksrc.install.mk:60: warning: overriding recipe for target '/spksrc/cross/pyephem/work/pyephem'
../../mk/spksrc.install.mk:48: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work/pyephem'
../../mk/spksrc.install.mk:80: warning: overriding recipe for target '/spksrc/cross/pyephem/work/.pyephem'
../../mk/spksrc.compile.mk:44: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work/.pyephem'
===>  Building package for arch armadaxp-5.1
make[1]: Entering directory '/spksrc/cross/pyephem'
../../mk/spksrc.depend.mk:56: warning: overriding recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.download.mk:147: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.checksum.mk:84: warning: overriding recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.depend.mk:56: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.extract.mk:58: warning: overriding recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.checksum.mk:84: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.patch.mk:62: warning: overriding recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.extract.mk:58: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.configure.mk:58: warning: overriding recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.patch.mk:62: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.compile.mk:44: warning: overriding recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.configure.mk:58: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.install.mk:60: warning: overriding recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/pyephem'
../../mk/spksrc.install.mk:48: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/pyephem'
../../mk/spksrc.install.mk:80: warning: overriding recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.compile.mk:44: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
===>  Set up toolchain 
make[2]: Nothing to be done for 'default'.
../../mk/spksrc.depend.mk:56: warning: overriding recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.download.mk:147: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.checksum.mk:84: warning: overriding recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.depend.mk:56: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.extract.mk:58: warning: overriding recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.checksum.mk:84: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.patch.mk:62: warning: overriding recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.extract.mk:58: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.configure.mk:58: warning: overriding recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.patch.mk:62: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.compile.mk:44: warning: overriding recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.configure.mk:58: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.install.mk:60: warning: overriding recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/pyephem'
../../mk/spksrc.install.mk:48: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/pyephem'
../../mk/spksrc.install.mk:80: warning: overriding recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
../../mk/spksrc.compile.mk:44: warning: ignoring old recipe for target '/spksrc/cross/pyephem/work-armadaxp-5.1/.pyephem'
===>  Installing for pyephem 
make[1]: Circular pre_install_target <- /spksrc/cross/pyephem/work-armadaxp-5.1/pyephem dependency dropped.
find /spksrc/cross/pyephem/work-armadaxp-5.1/install//usr/local/ \! -type d -printf '%P\n' | sort > .plist.tmp
/bin/sh: 1: cd: can't cd to /spksrc/cross/pyephem/work-armadaxp-5.1/pyephem
../../mk/spksrc.python-wheel.mk:28: recipe for target 'wheel_python_module' failed
make[1]: *** [wheel_python_module] Error 2
make[1]: Leaving directory '/spksrc/cross/pyephem'
../../mk/spksrc.cross-cc.mk:117: recipe for target 'arch-armadaxp-5.1' failed
make: [arch-armadaxp-5.1] Error 2 (ignored)

@Dr-Bean
Copy link
Contributor

Dr-Bean commented Apr 18, 2015

Not all Python modules are ready for wheels, and that might also be the case for pyephem. See the wiki page on the wheels subject for some suggestions (near the bottom of the page): https://github.com/SynoCommunity/spksrc/wiki/Using-wheels-to-distribute-Python-solutions. There might be some patching involved, but that's impossible to say without looking at the code itself.

@hansmbakker
Copy link
Author

I see. However, it seems that spksrc doesn't even download the sources (I also tried with spksrc.python-module.mk instead of spksrc.python-wheel.mk).

This makes it look as if the Makefile or another file is broken. Do you see anything strange in the pyephem files?

@Dr-Bean
Copy link
Contributor

Dr-Bean commented Apr 18, 2015

If the download doesn't even start, and with the errors you showed above, you're right that there must be a mistake or two in the Makefile.
Verify the package name, version and extension are correct in cross/pyephem/Makefile. I'd suggest to remove the unnecessary spaces there...;)
The wheel was created successfully after fixing that over here, so give that a try.

@Dr-Bean
Copy link
Contributor

Dr-Bean commented Apr 18, 2015

One addition: the current python_wheel.mk renames the cross-compiled wheels in this line: https://github.com/SynoCommunity/spksrc/blob/master/mk/spksrc.python-wheel.mk#L29. I'm not going into detail here, but that renaming action is specific to python2, because that's what the code was written for.

The issue you might run into is that during package installation, the python3 installer doesn't recognize the original name that the wheel ends up as (which is pyephem-3.7.5.3-cp34-cp34m-any.whl), and as a result, won't want to install that wheel. Check if that's the case.

If the wheel is not recognized, you can override the format by defining an INSTALL_TARGET in the pyephem Makefile, with as content the two lines in https://github.com/SynoCommunity/spksrc/blob/master/mk/spksrc.python-wheel.mk#L28,L29. You'll need to adjust L29 so that the wheel is renamed correctly from <module_name>-<version>-cp34-cp34m-any.whl to <module_name>-<version>-cp34-none-any.whl.
That will ultimately need to be fixed in python-wheel.mk obviously, but the above can be used as a temporary workaround.
In other words, make sure the wheel ends up as pyephem-3.7.5.3-cp34-none-any.whl

@hansmbakker
Copy link
Author

Great, thank you for thinking along!

I also stumbled upon pip not recognizing the wheels and decided building everything as modules for a working first build. Also I decided to comment out some optional modules, to try to first create a small version that actually runs.

I'm close now, I'll commit it when I make some progress :)

@Dr-Bean
Copy link
Contributor

Dr-Bean commented Apr 18, 2015

You mean pip doesn't recognize the cross-compiled wheels during creation of the package? That's the same issue. If you implement the approach I mentioned above, that should fix it.

@hansmbakker
Copy link
Author

I created a basic working version now at master...wind-rider:homeassistant

The server starts at port 8123 and works already with Philips hue and some python-based components.

The following still needs some attention for which I could use some assistance (expecially with the pid logic and log files):

  • Cross-compiled python modules are either added to python3 or temporarily excluded in requirements.txt
  • pid-file doesn't work properly yet, and probably because of that DSM doesn't see that it actually started
  • Log file does not work yet - the log file is being filled, but its contents are not shown in DSM

@Dr-Bean
Copy link
Contributor

Dr-Bean commented Apr 19, 2015

Try using busybox' start-stop-daemon instead of su, that might help with the pid approach.
Replace the tabs in dsmcontrol.sh, those shouldn't be there.

@hansmbakker
Copy link
Author

About https://github.com/SynoCommunity/spksrc/blob/master/mk/spksrc.python-wheel.mk#L29 that you mentioned - I see linux_i686 there.

I think that line doesn't work well here - I'm using your Docker environment in Ubuntu 14.04 64-bit. In the Docker file Debian Jessie 32 bit is defined, but still I see wheels like pyephem-3.7.5.3-cp27-none-linux_x86_64.whl in the wheelhouse folder.

Two questions:

  • can I add @rename -f 's/linux_x86_64/any/g' $(WORK_DIR)/wheelhouse/*.whl to spksrc.python-wheel.mk to solve it?
  • why is cp27 written there (or for some other wheels py2)?

@Dr-Bean
Copy link
Contributor

Dr-Bean commented Apr 19, 2015

Yes, that would work (it's essentially the same thing as I suggested before)
It's suboptimal whichever way you do it, because that part of the wheel filename depend on the host arch.
The wheel spec is currently being updated to 'understand' linux arches (at least, I hope that's one of the results), but that might take a bit.
Along with that, the rename command itself isn't portable, so that needs to be changed anyway.

@hansmbakker
Copy link
Author

I updated my comment above to make it more clear, but after that I saw that you already responded.

One more question - why is cp27 written there (or for some other wheels py2)? Should I worry about it (my package needs python3.4)

@Dr-Bean
Copy link
Contributor

Dr-Bean commented Apr 19, 2015

See https://www.python.org/dev/peps/pep-0427/#file-name-convention and https://www.python.org/dev/peps/pep-0425/#details for a bit of background. When (cross) compiling, the cp27 is shorthand for CPython version 2.7. cp34 is the shorthand notation for CPython version 3.4. I'm sure you can guess what py2 and py3 stand for ;)

@hansmbakker
Copy link
Author

Can it be that these py2 etc wheels are downloaded because of the PIP definition in spksrc.common.mk?

I think it won't be a problem as wheels are still downloaded by pip3.4 in installer.sh.

@Dr-Bean
Copy link
Contributor

Dr-Bean commented Apr 19, 2015

I could use a bit more context here...
You should get a cp27 tag when cross-compiling with Python2.7, and a cp34 tag when cross-compiling with Python3.4. I don't think it's even possible to get a cp27 tag when cross-compiling with Python3.4.
I don't think either tag can be installed on the other platform, that's what the tag is there for.

As for pure-python wheels: those end up with a py2 or py3 tag, which, amongst other things, depends on the pip that is used. You might run into installation issues when installing a py2 wheel on a py3 platform and vv, but you can easily test that.
Ideally, the name should probably end up with a py2.py3 tag, but that depends on the module, not spksrc.

As for how to 'force' a py3 tag instead of a py2 tag, you'll probably need to use python3's pip, instead of python2's pip. In other words, Python3 should be installed on the host, pip should be installed for that Python version, and common.mk should reference a pip3 variable, which can then be used during wheel creation. Or something like that, this is just off the top of my head.

@hansmbakker
Copy link
Author

What I meant was:
Some wheels are not built by me, but downloaded during the make process. In the case of pyephem, pip apparently downloads pyephem-3.7.5.3-cp27-none-linux_x86_64.whl alongside the crosscompiled pyephem-3.7.5.3-cp34-cp34m-any.whl.

I also think pip3 should be used to download python 3 wheels, but in spksrc.common.mk, the variable PIP is hardcoded to 'pip' instead of depending on the compilation.

Is this done on purpose?

@Dr-Bean
Copy link
Contributor

Dr-Bean commented Apr 19, 2015

To be clear, platform/python-specific wheels are never downloaded, those are always created locally.

The reason that you end up with two wheels is because Python2's pip doesn't recognize the cross-compiled wheel as valid, because Python2's pip only wants a cp27 tag, and rejects the cp34 tag on the existing wheel. So, it attempts to create a cp27 version of it, and that won't work, because it's not cross-compiled.
I believe I explained that before: the current wheel code was written for Python2, not Python3, so yes, adjustments need to be made.

@hansmbakker
Copy link
Author

@Dr-Bean, thank you for all your help :)

I managed to create a basic working version and created a pull request at #1655. I think we best continue discussing the code there, so I'll close this issue for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants