python: moving it to github; and 2.7 vs 3.4

[commodo]

Seems this package was not yet moved to Github.
I'm guessing it's because nobody wanted to, so far.
I can take maintainership of Python and friends (all Python packages), if nobody else wants to.

I was also thinking of starting a discussion on which version should the default Python version be.
Right now, it's 2.7, but we could look into making 3.4 the version when moving it to Github.

An idea would be to vote on it.
Evidently I'd vote for 3.4, but since I don't have a strong preference for it [other than it's newer and someday going to be the default] I decided to try to put it to a vote.

Thanks
Alex

[sbyx]

I'd probably go with 3.4 as well and yes its most likely that it isn't here because nobody felt compelled enough to become the maintainer. Also I guess we can take it slow and move the python modules one by one. No need to move everything at the same time. Also it would probably be easier if we grant you commit rights here, let me know what you think about that.

[commodo]

I'll respectfully refuse the commit rights.
Doing it by pull-requests is more than fine for me.

I sometimes find myself goofing sometimes due to multitasking and not having direct commit-rights helps me not mess up.
The way this would work well, is that I work in a branch and every once in a while I do a pull-request with some batch updates which I have tested on several targets.

Thanks

[sbyx]

Fine with me as well. Just wanted to make your life easier, but if you prefer PRs then that's fine as well.

[thess]

I would go with Python 3 if we choose one version going forward:

Python 2 vs Python 3: http://www.raspberrypi.org/documentation/usage/python/more.md

The short version: Python 2 is legacy, Python 3 is the present and future of the language.

Python 2 was released in 2000, and Python 3 was released in 2008. Python 3 is recommended, but some libraries have not yet been ported to Python 3 which is why Python 2 is still present and widely used.

You will probably need to test each imported library and/or applet to be sure it works with Python 3. I doubt we could just move the existing libraries/applets as is. Even though I am not much of a Python user, I would be happy to help with testing where possible.

[commodo]

Python 3 is the recommended version for forwardness, and has been for some time.
The general problem I've seen [and the reason for Python 2 still being the default] is legacy code and libraries, and the effort to port them to Python 3.

Seems [for OpenWRT] Python is not such a big deal yet, which means that making 3 the default will not be as difficult as I thought initially.

As for the libraries we'll see; some of them may not need to be imported, as they're already included [or have equivalents] in Python 3. For most, I hope the library creators did create Python 3 versions, and for the rest, it remains to be seen.

First thing to start with, will be the interpreter and we'll work up our way to other libs.

Thanks

[commodo]

Short update.
Just so that nobody thinks I timed-out.

Some commits can be found here : https://github.com/commodo/packages/commits/python-migration
At the moment, it compiles, but does not work.
Some libs are not copied on the target.

I'll have to comb through the files to see which should still be included, and which not.
But I think I'll throw in the whole Python beast in, and start stripping it into subpackages.

Thanks

[commodo]

Right.

So, I managed to get it working.
No libraries are added at the moment, will think about how to that.
Pull request is this: #396

I also have this commit here: commodo@abab13d
The idea is to try to allow 3 configurations for the main Python package, a Bare configuration (just the interpreter), a Full configuration (the entire Python beast) and something in the middle (popular choices).
Evidently, the idea is to also re-add the packages I threw out, as optional/installable packages.

A problem I have with the CONFIG_PYTHON_FULL/CONFIG_PYTHONBARE variables, is that it does not seem to update my firmware image if I switch from Bare to Full or vice-versa.
After a make clean [I haven't yet tried distclean] it seems to work, and it builds fine.
Any ideas ?
I'm suspecting some build stamps [for installed].

At the moment I am trying to evaluate the bloat of the Full installation mode.
The whole Python libraries are in the 34 MB [Python code] and around 4-5 MB shared libraries [depending on architecture].
I guess for a x86[_64] VM configuration this would be fine.
Not sure about real-world devices.
I have a PPC board that can handle a full Python installation.

Thoughts/Considerations if we should leave/add a Full Python build config flag ?
Reason I am asking, is that maybe someone knows about some usage stats about OpenWRT.

[jow-]

Try using

PKG_CONFIG_DEPENDS := CONFIG_PYTHON_FULL CONFIG_PYTHON_BARE

Make sure to declare that variable before including package.mk

[sbyx]

At first, thanks for your efforts here. However what reason is there to make this a config option (bare, mid, full) instead of making mid and full separate packages depending on bare + whatever set of .so's and .py files you packaged as python modules?

The reason is that external packages are auto-build but different config options aren't. So with your approach you would have exactly one python variant in the repositories with the packages-approach you could choose directly from the binary packages.

[commodo]

@sbyx
That was one approach I was thinking off [making bare/mid/full separate packages].
Then I thought about trying to compress the logic into a nicer menuconfig.
Doing the bare/mid/full approach is kind of like a presets mechanism; but I did cheat on the full preset right now, since the full preset just throws everything in.
So, if you'd select mid, then it would get bare + some of the more common/popular Python packages/modules automatically selected.
But I can understand now [actually I did not think of this up until now], that it would be more difficult to depend on a Python mid package/setup, if no Python mid setup is created.
In that case, you'd need to depend on all the Python packages you need to depend

Bouncing some these ideas off of you guys helps me decide easier, and stay on track with the whole guidelines.
I'll think about it some more. Maybe I'll disable that config think.

Thanks

[commodo]

Will reference this pull request : #429

So, python2 is back, and seems to stay around a while longer.
Will create a patch to remove it from the old packages.

Was thinking about how to package python (2 and 3), i.e. what should python-mini(mal) contain (well, at least for python3)

My idea is to take a look at the pypi ranking and package python-mini(mal) such that it supports the top 10 packages.

If anyone has ideas/thoughts/knowledge, please share.

Thanks
Alex

[sbyx]

pypi ranking is for external extensions not for the builtin ones. not sure how big the "standard library" of python actually is and what might be worth leaving out in -minimal.

Edit: also no need for patch to remove it from oldpackages. I just did that.

[commodo]

Thanks for removing it from oldpackages :)

Well, I said in a message earlier on this thread:

The whole Python libraries are in the 34 MB [Python code] and around 4-5 MB shared libraries [depending on architecture].

39-40 megs could be a bit much to leave it all in.
I guess some trimming could help, since some libs are graphic-libs (tk for example).

The idea of looking at external libraries, is to help select which standard libraries get selected into the python-mini(mal) package.
I mean, I don't know of another way right now to tell which standard libs to select.

[kostko]

It would seem that one of the latest updates (since the move from oldpackages) breaks build of Python modules that need to compile C code. The compilation tries to use the host's compiler and therefore generates invalid binaries or fails to compile.

This happens at least when building the following package with the latest python (reverting the python package to the old one fixes these issues):

https://github.com/wlanslovenija/firmware-packages-opkg/tree/master/lang/python-greenlet

[commodo]

Will check asap.

[hnyman]

python 2.7.8 seems to also have some missing deps for the ncurses module, says buildbot:
http://buildbot.openwrt.org:8010/broken_packages/ar71xx/python/compile.txt

Package python-ncurses is missing dependencies for the following libraries:
libncursesw.so.5
libpanelw.so.5
make[2]: *** [/mnt/dl/slave-8/ar71xx/build/bin/ar71xx/packages/packages/python-ncurses_2.7.8-3_ar71xx.ipk] Error 1
make[2]: Leaving directory `/mnt/dl/slave-8/ar71xx/build/feeds/packages/lang/python'

[commodo]

Hello,

@hnyman and @Hostko

Created this pull request : #445
This fixes both issues reported above.
Changes were in my branch yesterday, but I wanted to first try that they work.
I'm happy to say that they do.
Well, I tested with the python-greenlets module; it compiled and imported successfully on a ppc system.

@hnyman : sbyx told me about it yesterday: #438
Seems after Python 2.7.5, the build script (of Python) prefers ncursesw and if some other package installs it, Python will just pick it up, with no warning.

Please try again with this update and if all is good, I can probably start to update python3 the same way the python package was updated.

Thanks
Alex

[kostko]

I can confirm that the new Python package fixes the module build issues. Thanks @commodo!

[commodo]

@sbyx or @hnyman or @thess
I have this ticket opened for me : https://dev.openwrt.org/ticket/18185

So, there would be 2 possible fixes here:

1. Patch all (old)packages that have python-mini as dependency to use python, since they're both the same (and always were)
2. Add the python-mini package back; personally I think this would be a bit pointless. Maybe if there would be a way to "symlink" python-mini to python, it would be ok

And there would be a 3rd option: to ignore/close the ticket, since the issue is not that big.
It's mostly a warning rather than an error and it's in the old packages.

There are 24 old packages that reference python-mini and I've prepared patches for all of them (in the old packages).

What would you guys recommend ?

Thanks
Alex

[commodo]

Ah, just got my answer : https://dev.openwrt.org/ticket/18185#comment:2