Update python/hostpython recipes for Python 2.7.13 #242

d0mj · 2017-02-22T13:42:16Z

Version bumps for SDL2 (2.0.5), openssl (1.0.2k), python and hostpython (2.7.13) with changes to patches and build recipes.

…on (2.7.13) with appropriate changes to patches and build recipes.

d0mj · 2017-02-22T13:48:45Z

I've not tested any recipes beyond those pulled in by openssl and kivy. An archived app appears to make it through the automated App Store checks for internal beta testing on TestFlight and can be loaded onto real devices.

Zen-CODE · 2017-02-28T19:47:22Z

Thanks for this - there is a lot of work here. I'll do some testing myself and let you know.

Zen-CODE · 2017-03-01T16:57:55Z

Running OSX 10.10.5, XCode 7.2.

After a 'git clean -dxf', and checking out this branch, then running

./toolchain.py build kivy pyobjus openssl

I get the following:

rfc:kivy-ios camy$ ./toolchain.py build kivy pyobjus openssl
Want to build ['kivy', 'pyobjus', 'openssl']
Loaded recipe kivy (depends of ['python', 'sdl2', 'sdl2_image', 'sdl2_mixer', 'sdl2_ttf', 'ios', 'pyobjus'], optional are [])
Loaded recipe pyobjus (depends of ['python'], optional are [])
Loaded recipe openssl (depends of [], optional are [])
Loaded recipe python (depends of ['hostpython', 'libffi'], optional are ['openssl'])
Loaded recipe sdl2 (depends of [], optional are [])
Loaded recipe sdl2_image (depends of ['sdl2'], optional are [])
Loaded recipe sdl2_mixer (depends of ['sdl2'], optional are [])
Loaded recipe sdl2_ttf (depends of ['sdl2', 'freetype'], optional are [])
Loaded recipe ios (depends of ['python'], optional are [])
Loaded recipe hostpython (depends of ['hostlibffi'], optional are ['openssl'])
Loaded recipe libffi (depends of [], optional are [])
Loaded recipe freetype (depends of [], optional are [])
Loaded recipe hostlibffi (depends of [], optional are [])
Build order is ['freetype', 'hostlibffi', 'libffi', 'openssl', 'sdl2', 'hostpython', 'sdl2_image', 'sdl2_mixer', 'sdl2_ttf', 'python', 'ios', 'pyobjus', 'kivy']
Include dir added: {arch.arch}/freetype
Include dir added: {arch.arch}/hostlibffi
Include dir added: {arch.arch}/libffi
Include dir added: {arch.arch}/openssl
Include dir added: common/sdl2
Global: hostpython located at /Users/camy/Repos/kivy-ios/dist/hostpython/bin/python
Global: hostpgen located at /Users/camy/Repos/kivy-ios/dist/hostpython/bin/pgen
Include dir added: common/sdl2_image
Include dir added: common/sdl2_mixer
Include dir added: common/sdl2_ttf
# (ignored) Download freetype
# (ignored) Extract freetype
# (ignored) Build_all freetype
# (ignored) Download hostlibffi
# (ignored) Extract hostlibffi
Build hostlibffi for x86_64 (filtered)
# (ignored) Build hostlibffi
# (ignored) Download libffi
# (ignored) Extract libffi
# (ignored) Build_all libffi
# (ignored) Download openssl
# (ignored) Extract openssl
# (ignored) Build_all openssl
# (ignored) Download sdl2
# (ignored) Extract sdl2
# (ignored) Build_all sdl2
# (ignored) Download hostpython
# (ignored) Extract hostpython
Build_all hostpython
Build hostpython for x86_64 (filtered)
# (ignored) Build hostpython
Install include files for hostpython
# (ignored) Install_include hostpython
Install frameworks for hostpython
# (ignored) Install_frameworks hostpython
Install sources for hostpython
# (ignored) Install_sources hostpython
Install hostpython
ln: /Users/camy/Repos/kivy-ios/build/hostpython/x86_64/Python-2.7.13/Python/hostpython: File exists
Traceback (most recent call last):
  File "./toolchain.py", line 1319, in <module>
    ToolchainCL()
  File "./toolchain.py", line 1089, in __init__
    getattr(self, args.command)()
  File "./toolchain.py", line 1113, in build
    build_recipes(args.recipe, ctx)
  File "./toolchain.py", line 981, in build_recipes
    recipe.execute()
  File "/Users/camy/Repos/kivy-ios/toolchain.py", line 590, in execute
    self.build_all()
  File "/Users/camy/Repos/kivy-ios/toolchain.py", line 56, in _cache_execution
    f(self, *args, **kwargs)
  File "/Users/camy/Repos/kivy-ios/toolchain.py", line 704, in build_all
    self.install()
  File "/Users/camy/Repos/kivy-ios/recipes/hostpython/__init__.py", line 91, in install
    join(build_dir, "Python"))
  File "/Users/camy/Repos/kivy-ios/toolchain.py", line 39, in shprint
    for line in command(*args, **kwargs):
  File "./tools/external/sh.py", line 565, in next
    self.wait()
  File "./tools/external/sh.py", line 500, in wait
    self.handle_command_exit_code(exit_code)
  File "./tools/external/sh.py", line 516, in handle_command_exit_code
    raise exc(self.ran, self.process.stdout, self.process.stderr)
sh.ErrorReturnCode_1: 

  RAN: '/bin/ln -s /Users/camy/Repos/kivy-ios/build/hostpython/x86_64/Python-2.7.13/hostpython /Users/camy/Repos/kivy-ios/build/hostpython/x86_64/Python-2.7.13/Python'

  STDOUT:
ln: /Users/camy/Repos/kivy-ios/build/hostpython/x86_64/Python-2.7.13/Python/hostpython: File exists


  STDERR:

rfc:kivy-ios camy$ a

d0mj · 2017-03-01T21:03:40Z

Have you tried ./toolchain.py distclean prior to the build? I'm running macOS 10.12.3 and Xcode 8.2.1. I'll do some tests on 10.10.5 if you're still having this problem.

Zen-CODE · 2017-03-02T06:55:20Z

Still get an error. Full output here: https://gist.github.com/Zen-CODE/d515859d2d06fe02b8e7ad3c52c8b7b6

/Applications/Xcode.app/Contents/Developer/Toolchains/OSX10.10.xctoolchain/usr/bin/ranlib: file: libpython2.7.a(pymath.o) has no symbols
clang -Qunused-arguments -fcolor-diagnostics -lsqlite3 -lffi -L/Users/camy/Repos/kivy-ios/dist/hostlibffi/usr/local/lib -L/Users/camy/Repos/kivy-ios/dist/lib  -o python.exe \
			Modules/python.o \
			libpython2.7.a -ldl  -framework CoreFoundation -L/Users/camy/Repos/kivy-ios/dist/hostpython/lib -lz           -lssl -lcrypto             
ld: warning: directory not found for option '-L/Users/camy/Repos/kivy-ios/dist/hostpython/lib'
ld: in /Users/camy/Repos/kivy-ios/dist/lib/libcrypto.a(a_int.o), building for OSX, but linking in object file built for iOS, for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [python.exe] Error 1
Traceback (most recent call last):
  File "./toolchain.py", line 1319, in <module>
    ToolchainCL()
  File "./toolchain.py", line 1089, in __init__
    getattr(self, args.command)()
  File "./toolchain.py", line 1113, in build
    build_recipes(args.recipe, ctx)
  File "./toolchain.py", line 981, in build_recipes
    recipe.execute()
  File "/Users/camy/Repos/kivy-ios/toolchain.py", line 590, in execute
    self.build_all()
  File "/Users/camy/Repos/kivy-ios/toolchain.py", line 56, in _cache_execution
    f(self, *args, **kwargs)
  File "/Users/camy/Repos/kivy-ios/toolchain.py", line 704, in build_all
    self.install()
  File "/Users/camy/Repos/kivy-ios/recipes/hostpython/__init__.py", line 95, in install
    _env=build_env)
  File "/Users/camy/Repos/kivy-ios/toolchain.py", line 39, in shprint
    for line in command(*args, **kwargs):
  File "./tools/external/sh.py", line 565, in next
    self.wait()
  File "./tools/external/sh.py", line 500, in wait
    self.handle_command_exit_code(exit_code)
  File "./tools/external/sh.py", line 516, in handle_command_exit_code
    raise exc(self.ran, self.process.stdout, self.process.stderr)
sh.ErrorReturnCode_2: 

  RAN: '/usr/bin/make -C /Users/camy/Repos/kivy-ios/build/hostpython/x86_64/Python-2.7.13 bininstall inclinstall'

  STDOUT:
Makefile:1526: warning: overriding commands for target `Modules/_math.o'
Makefile:551: warning: ignoring old commands for target `Modules/_math.o'
Makefile:1619: warning: overriding commands for target `Modules/posixmodule.o'
Makefile:1508: warning: ignoring old commands for target `Modules/posixmodule.o'
Makefile:1620: warning: overriding commands for target `Modules/posixmodule.so'
Makefile:1509: warning: ignoring old commands for target `Modules/posixmodule.so'
Makefile:1621: warning: overriding commands for target `Modules/errnomodule.o'
Makefile:1510: warning: ignoring old commands for target `Modules/errnomodule.o'
Makefile:1622: warning: overriding commands for target `Modules/errnomodule.so'
Makefile:1511: warning: ignoring old comm... (81764 more, please see e.stdout)

  STDERR:

d0mj · 2017-03-06T21:54:21Z

This seems to be down to my hasty workaround for the OpenSSL dependancy of the updated hostpython. It probably shouldn't have worked at all, and clearly doesn't work on all versions of macOS. I can think of a couple of ways to do it properly.

Depend on having OpenSSL installed via homebrew/macports or the like.
Create a "hostopenssl" recipe.

I favour option 2 as it avoids any potential version mismatch issues. I'll see what I can whip up when I have some spare minutes.

Zen-CODE · 2017-03-07T03:13:33Z

Yes, I agree option 2 would be preferable. And any minutes you can whip together are appreciated. Getting these changes into master would be really nice...:-)

d0mj · 2017-03-08T22:26:11Z

Hopefully this works. hostopenssl should be pulled in if openssl is requested.

Zen-CODE · 2017-03-09T09:25:08Z

Pulled latest changes, 'git clean -dxf', './toolchain.py kivy openssl pyobjus', gives the error below. Please ask if you need more.

Compiling /Users/camy/Repos/kivy-ios/dist/root/python/lib/python2.7/xml/sax/saxutils.py ...
Compiling /Users/camy/Repos/kivy-ios/dist/root/python/lib/python2.7/xml/sax/xmlreader.py ...
Compiling /Users/camy/Repos/kivy-ios/dist/root/python/lib/python2.7/xmllib.py ...
Compiling /Users/camy/Repos/kivy-ios/dist/root/python/lib/python2.7/xmlrpclib.py ...
Compiling /Users/camy/Repos/kivy-ios/dist/root/python/lib/python2.7/zipfile.py ...
make: *** [libinstall] Error 1
Traceback (most recent call last):
  File "./toolchain.py", line 1319, in <module>
    ToolchainCL()
  File "./toolchain.py", line 1089, in __init__
    getattr(self, args.command)()
  File "./toolchain.py", line 1113, in build
    build_recipes(args.recipe, ctx)
  File "./toolchain.py", line 981, in build_recipes
    recipe.execute()
  File "/Users/camy/Repos/kivy-ios/toolchain.py", line 590, in execute
    self.build_all()
  File "/Users/camy/Repos/kivy-ios/toolchain.py", line 56, in _cache_execution
    f(self, *args, **kwargs)
  File "/Users/camy/Repos/kivy-ios/toolchain.py", line 704, in build_all
    self.install()
  File "/Users/camy/Repos/kivy-ios/recipes/python/__init__.py", line 82, in install
    _env=build_env)
  File "/Users/camy/Repos/kivy-ios/toolchain.py", line 39, in shprint
    for line in command(*args, **kwargs):
  File "./tools/external/sh.py", line 565, in next
    self.wait()
  File "./tools/external/sh.py", line 500, in wait
    self.handle_command_exit_code(exit_code)
  File "./tools/external/sh.py", line 516, in handle_command_exit_code
    raise exc(self.ran, self.process.stdout, self.process.stderr)
sh.ErrorReturnCode_2: 

  RAN: '/usr/bin/make -C /Users/camy/Repos/kivy-ios/build/python/i386/Python-2.7.13 install CROSS_COMPILE_TARGET=yes HOSTPYTHON=/Users/camy/Repos/kivy-ios/dist/hostpython/bin/python prefix=/Users/camy/Repos/kivy-ios/dist/root/python'

  STDOUT:
Makefile:1513: warning: overriding commands for target `Modules/_math.o'
Makefile:551: warning: ignoring old commands for target `Modules/_math.o'
Creating directory /Users/camy/Repos/kivy-ios/dist/root/python/bin
Creating directory /Users/camy/Repos/kivy-ios/dist/root/python/lib
/usr/bin/install -c python.exe /Users/camy/Repos/kivy-ios/dist/root/python/bin/python2.7
if test -f libpython2.7.a; then \
		if test -n "" ; then \
			/usr/bin/install -c -m 555  /Users/camy/Repos/kivy-ios/dist/root/python/bin; \
		else \
			/usr/bin/install -c -m 555 libpython2.7.a /Users/camy/Repos/kivy-ios/dist/root/python/lib/libpython2.7.a; \
			if test libpython2.7.a != libpython2.7.a; then \
				(cd /Users/camy/Repos/kivy-ios/dist/root/python/lib; ln -sf lib... (405580 more, please see e.stdout)

  STDERR:

Zen-CODE · 2017-03-09T10:00:49Z

A second run gets me further.

Compiling /Users/camy/Repos/kivy-ios/dist/root/python/lib/python2.7/xml/sax/xmlreader.py ...
Compiling /Users/camy/Repos/kivy-ios/dist/root/python/lib/python2.7/xmllib.py ...
Compiling /Users/camy/Repos/kivy-ios/dist/root/python/lib/python2.7/xmlrpclib.py ...
Compiling /Users/camy/Repos/kivy-ios/dist/root/python/lib/python2.7/zipfile.py ...
make: *** [libinstall] Error 1
Traceback (most recent call last):
  File "./toolchain.py", line 1319, in <module>
    ToolchainCL()
  File "./toolchain.py", line 1089, in __init__
    getattr(self, args.command)()
  File "./toolchain.py", line 1113, in build
    build_recipes(args.recipe, ctx)
  File "./toolchain.py", line 981, in build_recipes
    recipe.execute()
  File "/Users/camy/Repos/kivy-ios/toolchain.py", line 590, in execute
    self.build_all()
  File "/Users/camy/Repos/kivy-ios/toolchain.py", line 56, in _cache_execution
    f(self, *args, **kwargs)
  File "/Users/camy/Repos/kivy-ios/toolchain.py", line 704, in build_all
    self.install()
  File "/Users/camy/Repos/kivy-ios/recipes/python/__init__.py", line 82, in install
    _env=build_env)
  File "/Users/camy/Repos/kivy-ios/toolchain.py", line 39, in shprint
    for line in command(*args, **kwargs):
  File "./tools/external/sh.py", line 565, in next
    self.wait()
  File "./tools/external/sh.py", line 500, in wait
    self.handle_command_exit_code(exit_code)
  File "./tools/external/sh.py", line 516, in handle_command_exit_code
    raise exc(self.ran, self.process.stdout, self.process.stderr)
sh.ErrorReturnCode_2: 

  RAN: '/usr/bin/make -C /Users/camy/Repos/kivy-ios/build/python/i386/Python-2.7.13 install CROSS_COMPILE_TARGET=yes HOSTPYTHON=/Users/camy/Repos/kivy-ios/dist/hostpython/bin/python prefix=/Users/camy/Repos/kivy-ios/dist/root/python'

  STDOUT:
Makefile:1513: warning: overriding commands for target `Modules/_math.o'
Makefile:551: warning: ignoring old commands for target `Modules/_math.o'
/usr/bin/install -c python.exe /Users/camy/Repos/kivy-ios/dist/root/python/bin/python2.7
if test -f libpython2.7.a; then \
		if test -n "" ; then \
			/usr/bin/install -c -m 555  /Users/camy/Repos/kivy-ios/dist/root/python/bin; \
		else \
			/usr/bin/install -c -m 555 libpython2.7.a /Users/camy/Repos/kivy-ios/dist/root/python/lib/libpython2.7.a; \
			if test libpython2.7.a != libpython2.7.a; then \
				(cd /Users/camy/Repos/kivy-ios/dist/root/python/lib; ln -sf libpython2.7.a libpython2.7.a) \
			fi \
		fi; \
	else	true; \
	fi
running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or ... (398671 more, please see e.stdout)

  STDERR:

d0mj · 2017-03-10T16:46:27Z

Unfortunately I've not been able to reproduce your problem on 10.10.5. Clearly there's a difference in our configuration. What do you get from xcodebuild -version, libtool -V, clang --version, and any other tools that may be relevant?

Zen-CODE · 2017-03-11T04:44:14Z

The last error seems to indicate the zlib is try to load Tcl/Tk and fails. Trying to load Tkinter on that makes dies and closes Python. Reading up, it seems Tkinter is very problematic on Mac. Could that be the cause? I'm trying to get Tkinter running on that mac and will let you know.

Am I on the right track?

d0mj · 2017-03-11T11:04:15Z

Possibly, although I still think it's worth figuring out why a build works for me on a clean, up-to-date OSX 10.10.5 but not for you. Feel free to follow up on the Tkinter idea. I'm away from a computer for a couple of days so will pick up when I'm back.

Zen-CODE · 2017-03-13T20:08:19Z

Firstly, thanks for the PR. There are some changes here that are really nice.

But there is just too much happening here for us to merge safely into master. Too many things being changed that cannot be safely tested in isolation. It's our obligation to users to prevent undesired side effects.

Please could we ask that you separate these into independent PR"s that address one issue at a time. Atomic merges are ideal: merges that address one issue cleanly and clearly. They are not only less error prone, but make tracking errors down the line much easier.

So, if this PR could be separated into a series of PR"s that introduce the suggested changes one, test-able step at a time, we would be grateful .e.g. "Upgrade Python to 2.7.13", then "Upgrade openssl (1.0.2k)" etc. This would also make it much easier to see where strange errors creep in e.g. the tcl/tk issue.

Version bumps for SDL2 (2.0.5), openssl (1.0.2k), python and hostpyth…

dee96ac

…on (2.7.13) with appropriate changes to patches and build recipes.

Add hostopenssl recipe and fixes

0bf2527

Zen-CODE closed this Mar 13, 2017

akinwale added a commit to lbryio/kivy-ios that referenced this pull request Jun 1, 2018

Python 2.7.13 support based on kivy#242

45b97ea

Update python/hostpython recipes for Python 2.7.13 #242

Update python/hostpython recipes for Python 2.7.13 #242

d0mj commented Feb 22, 2017

d0mj commented Feb 22, 2017

Zen-CODE commented Feb 28, 2017

Zen-CODE commented Mar 1, 2017 •

edited

d0mj commented Mar 1, 2017

Zen-CODE commented Mar 2, 2017

d0mj commented Mar 6, 2017

Zen-CODE commented Mar 7, 2017

d0mj commented Mar 8, 2017

Zen-CODE commented Mar 9, 2017

Zen-CODE commented Mar 9, 2017

d0mj commented Mar 10, 2017

Zen-CODE commented Mar 11, 2017

d0mj commented Mar 11, 2017

Zen-CODE commented Mar 13, 2017

Update python/hostpython recipes for Python 2.7.13 #242

Update python/hostpython recipes for Python 2.7.13 #242

Conversation

d0mj commented Feb 22, 2017

d0mj commented Feb 22, 2017

Zen-CODE commented Feb 28, 2017

Zen-CODE commented Mar 1, 2017 • edited

d0mj commented Mar 1, 2017

Zen-CODE commented Mar 2, 2017

d0mj commented Mar 6, 2017

Zen-CODE commented Mar 7, 2017

d0mj commented Mar 8, 2017

Zen-CODE commented Mar 9, 2017

Zen-CODE commented Mar 9, 2017

d0mj commented Mar 10, 2017

Zen-CODE commented Mar 11, 2017

d0mj commented Mar 11, 2017

Zen-CODE commented Mar 13, 2017

Zen-CODE commented Mar 1, 2017 •

edited