buildozer uses wrong python version and disrespects requirement versions

[joergbrech]

Versions

• Python: 3.6 (and curiously 3.7, see below)
• OS: Ubuntu 16.04.6 LTS
• Buildozer: 1.11.1

Description

First of all, thanks for the really awesome project! I wrote a Kivy app two years ago and tried to revive the project, but am currently stuck on building the apk. I am running into two build errors I do not quite understand.

1/2 Buildozer does not use systemwide python.

Buildozer seems to download hostpython3 and uses this python distribution to build my dependencies, but the hostpython3 distribution does not seem to have cython installed

[INFO]:    -> directory context /data/tools/mapfix/.buildozer/android/platform/build/build/other_builds/pyproj-python3/armeabi-v7a__ndk_target_26/pyproj
[INFO]:    -> running python -c import sys; print(sys.path)
[INFO]:    Trying first build of pyproj to get cython files: this is expected to fail                                                                                      
[INFO]:    -> running python setup.py build_ext -v
Exception in thread background thread for pid 3764:                                                                                                                        
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/jan/.local/lib/python3.6/site-packages/sh.py", line 1540, in wrap
    fn(*args, **kwargs)
  File "/home/jan/.local/lib/python3.6/site-packages/sh.py", line 2459, in background_thread
    handle_exit_code(exit_code)
  File "/home/jan/.local/lib/python3.6/site-packages/sh.py", line 2157, in fn
    return self.command.handle_command_exit_code(exit_code)
  File "/home/jan/.local/lib/python3.6/site-packages/sh.py", line 815, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_1: 

  RAN: /data/tools/mapfix/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/python setup.py build_ext -v

  STDOUT:
ERROR: Cython.Build.cythonize not found. Cython is required to build from a repo.

The systemwide python has cython installed:

jan@jan-notebook:~/data/tools/mapfix$ which python
/usr/bin/python
jan@jan-notebook:~/data/tools/mapfix$ python --version
Python 3.6.8
jan@jan-notebook:~/data/tools/mapfix$ python
Python 3.6.8 (default, Oct  7 2019, 12:59:55) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import Cython
>>> print(Cython.__version__)
0.29.13

The one used by buildozer does not:

jan@jan-notebook:~/data/tools/mapfix$ /data/tools/mapfix/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/python --version
Python 3.7.1
jan@jan-notebook:~/data/tools/mapfix$ /data/tools/mapfix/.buildozer/android/platform/build/build/other_builds/hostpython3/desktop/hostpython3/native-build/python
Python 3.7.1 (default, Oct 23 2019, 21:24:49) 
[GCC 7.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import Cython
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'Cython'

2/2 Buildozer does not respect all version specifications in my requirements.

I have the line

requirements = python3,kivy,piexif,pillow,unidecode,exifread,numpy,pyproj==1.9.6,plyer

in my buildozer.spec but buildozer installs the latest version 2.4.0 of pyproj, which I know does not work:

[INFO]:    Downloading pyproj
[INFO]:    -> running mkdir -p /data/tools/mapfix/.buildozer/android/platform/build/packages/pyproj
[INFO]:    -> directory context /data/tools/mapfix/.buildozer/android/platform/build/packages/pyproj
[INFO]:    -> running basename https://github.com/jswhit/pyproj/archive/master.zip
[INFO]:    -> running rm -f .mark-master.zip                                                                                                                               
[INFO]:    Downloading pyproj from https://github.com/jswhit/pyproj/archive/master.zip
[INFO]:    -> running touch .mark-master.zip
[INFO]:    <- directory context /data/tools/mapfix/.buildozer/android/platform/python-for-android

Version 2.4.0 verified via:

jan@jan-notebook:~/data/tools/mapfix$ cd /data/tools/mapfix/.buildozer/android/platform/build/build/other_builds/pyproj-python3/armeabi-v7a__ndk_target_26/pyproj
jan@jan-notebook:/data/tools/mapfix/.buildozer/android/platform/build/build/other_builds/pyproj-python3/armeabi-v7a__ndk_target_26/pyproj$ grep '__version__' pyproj/__init__.py 
__version__ = "2.4.0"

Notes

I also have the job set up in a clean container on Travis. You can find the log here and the travis config here. Thank you for looking into this!

buildozer.spec

Command:

buildozer android debug

Spec file:

[app]

# (str) Title of your application
title = MapFix

# (str) Package name
package.name = MapFix

# (str) Package domain (needed for android/ios packaging)
package.domain = jan.kleinert

# (str) Source code where the main.py live
source.dir = mapfix

# (list) Source files to include (let empty to include all the files)
source.include_exts = py,png,jpg,kv,atlas,json

# (list) List of inclusions using pattern matching
#source.include_patterns = assets/*,images/*.png

# (list) Source files to exclude (let empty to not exclude anything)
source.exclude_exts = spec,sh

# (list) List of directory to exclude (let empty to not exclude anything)
source.exclude_dirs = tests, bin
#source.exclude_dirs = tests, bin, maps

# (list) List of exclusions using pattern matching
#source.exclude_patterns = license,images/*/*.jpg

# (str) Application versioning (method 1)
# version = 0.1.1

# (str) Application versioning (method 2)
version.regex = __version__ = ['"](.*)['"]
version.filename = %(source.dir)s/__init__.py

# (list) Application requirements
# comma seperated e.g. requirements = sqlite3,kivy
requirements = python3,kivy,piexif,pillow,unidecode,exifread,numpy,pyproj==1.9.3,plyer

# (str) Custom source folders for requirements
# Sets custom source for any requirements with recipes
# requirements.source.kivy = ../../kivy

# (list) Garden requirements
#garden_requirements =

# (str) Presplash of the application
presplash.filename = %(source.dir)s/data/images/presplash.png

# (str) Icon of the application
icon.filename = %(source.dir)s/data/images/logo_mapfix.png

# (str) Supported orientation (one of landscape, portrait or all)
orientation = portrait

# (list) List of service to declare
#services = NAME:ENTRYPOINT_TO_PY,NAME2:ENTRYPOINT2_TO_PY

#
# OSX Specific
#

#
# author = © Copyright Info

#
# Android specific
#

# (bool) Indicate if the application should be fullscreen or not
fullscreen = 1

# (list) Permissions
android.permissions = INTERNET,ACCESS_FINE_LOCATION,ACCESS_COARSE_LOCATION,WRITE_EXTERNAL_STORAGE, READ_EXTERNAL_STORAGE, CAMERA

# (int) Android API to use
#android.api = 19

# (int) Minimum API required
android.minapi = 26

# (int) Android SDK version to use
# android.sdk = 20
# android.sdk = 24

# (str) Android NDK version to use
# android.ndk = 10.3.2

# (bool) Use --private data storage (True) or --dir public storage (False)
#android.private_storage = True

# (str) Android NDK directory (if empty, it will be automatically downloaded.)
# android.ndk_path = /home/jan/tools/crystax/crystax-ndk-10.3.2/

# (str) Android SDK directory (if empty, it will be automatically downloaded.)
#android.sdk_path =

# (str) ANT directory (if empty, it will be automatically downloaded.)
#android.ant_path =

# (str) python-for-android git clone directory (if empty, it will be automatically cloned from github)
#android.p4a_dir =

# (list) python-for-android whitelist
#android.p4a_whitelist =

# (bool) If True, then skip trying to update the Android sdk
# This can be useful to avoid excess Internet downloads or save time
# when an update is due and you just want to test/build your package
# android.skip_update = False

# (str) Android entry point, default is ok for Kivy-based app
#android.entrypoint = org.renpy.android.PythonActivity

# (list) List of Java .jar files to add to the libs so that pyjnius can access
# their classes. Don't add jars that you do not need, since extra jars can slow
# down the build process. Allows wildcards matching, for example:
# OUYA-ODK/libs/*.jar
#android.add_jars = foo.jar,bar.jar,path/to/more/*.jar

# (list) List of Java files to add to the android project (can be java or a
# directory containing the files)
#android.add_src =

# (str) python-for-android branch to use, if not master, useful to try
# not yet merged features.
#android.branch = master

# (str) OUYA Console category. Should be one of GAME or APP
# If you leave this blank, OUYA support will not be enabled
#android.ouya.category = GAME

# (str) Filename of OUYA Console icon. It must be a 732x412 png image.
#android.ouya.icon.filename = %(source.dir)s/data/ouya_icon.png

# (str) XML file to include as an intent filters in <activity> tag
#android.manifest.intent_filters =

# (list) Android additionnal libraries to copy into libs/armeabi
#android.add_libs_armeabi = libs/android/*.so
#android.add_libs_armeabi_v7a = libs/android-v7/*.so
#android.add_libs_x86 = libs/android-x86/*.so
#android.add_libs_mips = libs/android-mips/*.so

# (bool) Indicate whether the screen should stay on
# Don't forget to add the WAKE_LOCK permission if you set this to True
#android.wakelock = False

# (list) Android application meta-data to set (key=value format)
#android.meta_data =

# (list) Android library project to add (will be added in the
# project.properties automatically.)
#android.library_references =

# (str) Android logcat filters to use
android.logcat_filters = *:S python:D

# (bool) Copy library instead of making a libpymodules.so
#android.copy_libs = 1

# Set the backend to pygame (rather than sdl2)
#p4a.bootstrap = pygame
#p4a.source_dir= ~/data/tools/python-for-android

# (bool) If True, then automatically accept SDK license
# agreements. This is intended for automation only. If set to False,
# the default, you will be shown the license when first running
# buildozer.
android.accept_sdk_license = True

#
# iOS specific
#

# (str) Path to a custom kivy-ios folder
#ios.kivy_ios_dir = ../kivy-ios

# (str) Name of the certificate to use for signing the debug version
# Get a list of available identities: buildozer ios list_identities
#ios.codesign.debug = "iPhone Developer: <lastname> <firstname> (<hexstring>)"

# (str) Name of the certificate to use for signing the release version
#ios.codesign.release = %(ios.codesign.debug)s


[buildozer]

# (int) Log level (0 = error only, 1 = info, 2 = debug (with command output))
log_level = 2

# (int) Display warning if buildozer is run as root (0 = False, 1 = True)
warn_on_root = 1

# (str) Path to build artifact storage, absolute or relative to spec file
# build_dir = ./.buildozer

# (str) Path to build output (i.e. .apk, .ipa) storage
# bin_dir = ./bin

#    -----------------------------------------------------------------------------
#    List as sections
#
#    You can define all the "list" as [section:key].
#    Each line will be considered as a option to the list.
#    Let's take [app] / source.exclude_patterns.
#    Instead of doing:
#
#[app]
#source.exclude_patterns = license,data/audio/*.wav,data/images/original/*
#
#    This can be translated into:
#
#[app:source.exclude_patterns]
#license
#data/audio/*.wav
#data/images/original/*
#


#    -----------------------------------------------------------------------------
#    Profiles
#
#    You can extend section / key with a profile
#    For example, you want to deploy a demo version of your application without
#    HD content. You could first change the title to add "(demo)" in the name
#    and extend the excluded directories to remove the HD content.
#
#[app@demo]
#title = My Application (demo)
#
#[app:source.exclude_patterns@demo]
#images/hd/*
#
#    Then, invoke the command line with the "demo" profile:
#
#buildozer --profile demo android debug

Logs

You can find the buildozer build logs on travis. Let me know if you want me to attach a complete log here anyway.

[tito]

Hi, Please don't open issue for multiple issue, it is hard to track down everything, and cannot close half of the opened issue :)

First of all, in cross compilation, host != target. Host is your computer, target is the final device (android device, running on ARM mostly).
When cross compiling, we compile our own python for host and target, to have matching version and be able to run locally some setup.py that will match the target version:

• python-for-android don't allow to change the python3 version. Because there is patches associated to the python3 recipes, it would require us to track all the Python 3 version associated with their patch for android.
• Cython is used manually for various reason (biglinking), and is not installed on the hostpython so this is expected.

As for the recipe not accepting your version, it may be because the recipe doesn't allow it. Check the recipe on python-for-android as well.

If you want to talk more about the compilation, come to the discord, and check python-for-android project. Buildozer is just a wrapper around it.

[support]

👋 @joergbrech, we use the issue tracker exclusively for bug reports and feature requests. However, this issue appears to be a support request. Please use our support channels to get help with the project.

Let us know if this comment was made in error, and we'll be happy to reopen the issue.