Skip to content

SciPy recipe fails with current development branch of P4A #3116

New issue
New issue
Open
#3136
Open
SciPy recipe fails with current development branch of P4A#3116
#3136
Labels
recipe
@mriscoc

Description

@mriscoc
mriscoc
opened on Mar 5, 2025
Contributor

WSL2 on Windows 11
Ubuntu 22.04.5 LTS
Buildozer 1.5.1.dev0
$LEGACY_NDK: android-ndk-r21e with fortran addons

Relevant configuration lines for buildozer.spec:

requirements = python3,kivy,scipy
android.minapi = 24
android.archs = arm64-v8a
p4a.branch = develop
log_level = 2

Minimal Kivy test:

import kivy
import scipy

from kivy.app import App
from kivy.uix.button import Button

def callback(instance):
    scipy.show_config()

class MyApp(App):
  def build(self):
    btn1 = Button(text='SciPy show config', size=(100, 50))
    btn1.bind(on_press=callback)
    return btn1

if __name__ == '__main__':
    MyApp().run()

Issue:

buildozer -v android debug finished with the error: 'int_t' is not a type identifier

[INFO]:    Building scipy for arm64-v8a
[INFO]:    scipy apparently isn't already in site-packages
[DEBUG]:   -> running pip install numpy --target /home/mrisco/kivy/test_scipy/.buildozer/android/platform/build-arm64-v8a/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages --python-version 3.11.5 --only-binary=:all: --upgrade
[DEBUG]:        Collecting numpy
[DEBUG]:          Using cached numpy-2.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (62 kB)
[DEBUG]:        Using cached numpy-2.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.4 MB)
[DEBUG]:        Installing collected packages: numpy
[DEBUG]:        Successfully installed numpy-2.2.3
[INFO]:    Building compiled components in scipy
[INFO]:    -> directory context /home/mrisco/kivy/test_scipy/.buildozer/android/platform/build-arm64-v8a/build/other_builds/scipy/arm64-v8a__ndk_target_24/scipy
[DEBUG]:   -> running python3 setup.py build_ext -v -j 12
[DEBUG]:        /home/mrisco/kivy/test_scipy/.buildozer/android/platform/build-arm64-v8a/build/other_builds/scipy/arm64-v8a__ndk_target_24/scipy/setup.py:136: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
[DEBUG]:          from distutils.command.sdist import sdist
[DEBUG]:        /home/mrisco/kivy/test_scipy/.buildozer/android/platform/build-arm64-v8a/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/_distutils_hack/__init__.py:11: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils.
[DEBUG]:          warnings.warn(
[DEBUG]:        /home/mrisco/kivy/test_scipy/.buildozer/android/platform/build-arm64-v8a/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/_distutils_hack/__init__.py:26: UserWarning: Setuptools is replacing distutils.
[DEBUG]:          warnings.warn("Setuptools is replacing distutils.")
[DEBUG]:        /home/mrisco/kivy/test_scipy/.buildozer/android/platform/build-arm64-v8a/build/other_builds/scipy/arm64-v8a__ndk_target_24/scipy/setup.py:519: DeprecationWarning:
[DEBUG]:   
[DEBUG]:          `numpy.distutils` is deprecated since NumPy 1.23.0, as a result
[DEBUG]:          of the deprecation of `distutils` itself. It will be removed for
[DEBUG]:          Python >= 3.12. For older Python versions it will remain present.
[DEBUG]:          It is recommended to use `setuptools < 60.0` for those Python versions.
[DEBUG]:          For more details, see:
[DEBUG]:            https://numpy.org/devdocs/reference/distutils_status_migration.html
[DEBUG]:   
[DEBUG]:   
[DEBUG]:          from numpy.distutils.core import setup
[DEBUG]:        Cythonizing sources
[DEBUG]:        Running scipy/stats/_generate_pyx.py
[DEBUG]:        Running scipy/linalg/_generate_pyx.py
[DEBUG]:        Running scipy/special/_generate_pyx.py
[DEBUG]:        Processing scipy/ndimage/src/_cytest.pyx
[DEBUG]:        Processing scipy/ndimage/src/_ni_label.pyx
[DEBUG]:        Processing scipy/stats/_stats.pyx
[DEBUG]:        Processing scipy/stats/_biasedurn.pyx
[DEBUG]:        Processing scipy/stats/_qmc_cy.pyx
[DEBUG]:        Processing scipy/stats/_sobol.pyx
[DEBUG]:        Processing scipy/stats/_unuran/unuran_wrapper.pyx
[DEBUG]:        Processing scipy/stats/_levy_stable/levyst.pyx
[DEBUG]:        Processing scipy/stats/_boost/src/skewnorm_ufunc.pyx
[DEBUG]:        Processing scipy/stats/_boost/src/beta_ufunc.pyx
[DEBUG]:        Processing scipy/stats/_boost/src/binom_ufunc.pyx
[DEBUG]:        Processing scipy/stats/_boost/src/invgauss_ufunc.pyx
[DEBUG]:        Processing scipy/stats/_boost/src/ncx2_ufunc.pyx
[DEBUG]:        Processing scipy/stats/_boost/src/hypergeom_ufunc.pyx
[DEBUG]:        Processing scipy/stats/_boost/src/nbinom_ufunc.pyx
[DEBUG]:        Processing scipy/stats/_boost/src/ncf_ufunc.pyx
[DEBUG]:        Processing scipy/stats/_boost/src/nct_ufunc.pyx
[DEBUG]:        Processing scipy/stats/_rcont/rcont.pyx
[DEBUG]:        Processing scipy/sparse/_csparsetools.pyx.in
[DEBUG]:   
[DEBUG]:        Error compiling Cython file:
[DEBUG]:        ------------------------------------------------------------
[DEBUG]:        ...
[DEBUG]:                        needs_self_labeling = True
[DEBUG]:   
[DEBUG]:                        # Take neighbor labels
[DEBUG]:                        PyArray_ITER_RESET(itstruct)
[DEBUG]:                        for ni in range(num_neighbors):
[DEBUG]:                            neighbor_use_prev = (<np.int_t *> PyArray_ITER_DATA(itstruct))[0]
[DEBUG]:                                                 ^
[DEBUG]:        ------------------------------------------------------------
[DEBUG]:   
[DEBUG]:        _ni_label.pyx:329:42: 'int_t' is not a type identifier

Activity

kuzeyron
added
recipe
on Mar 5, 2025
mriscoc

mriscoc commented on Mar 7, 2025

@mriscoc
mriscoc
on Mar 7, 2025
ContributorAuthor

Current recipe uses SciPy maintenance/1.11.3:

python-for-android/pythonforandroid/recipes/scipy/__init__.py

Lines 18 to 20 in 9b0eed1

version = 'maintenance/1.11.x'
url = 'git+https://github.com/scipy/scipy.git'
git_commit = 'b430bf54b5064465983813e2cfef3fcb86c3df07' # version 1.11.3

which uses a deprecated Numpy type np.int_t:
https://github.com/scipy/scipy/blob/1d3a067c2ccd0a6efddeb3194163aa9a3879d26e/scipy/ndimage/src/_ni_label.pyx#L328-L331

                for ni in range(num_neighbors):
                    neighbor_use_prev = (<np.int_t *> PyArray_ITER_DATA(itstruct))[0]
                    neighbor_use_adjacent = (<np.int_t *> (<char *> PyArray_ITER_DATA(itstruct) + ss))[0]
                    neighbor_use_next = (<np.int_t *> (<char *> PyArray_ITER_DATA(itstruct) + 2 * ss))[0]

That was fixed in the SciPy version 1.12.0:
https://github.com/scipy/scipy/blob/4edfcaa3ce8a387450b6efce968572def71be089/scipy/ndimage/src/_ni_label.pyx#L328-L331

                for ni in range(num_neighbors):
                    neighbor_use_prev = (<np.npy_bool *> PyArray_ITER_DATA(itstruct))[0]
                    neighbor_use_adjacent = (<np.npy_bool *> (<char *> PyArray_ITER_DATA(itstruct) + ss))[0]
                    neighbor_use_next = (<np.npy_bool *> (<char *> PyArray_ITER_DATA(itstruct) + 2 * ss))[0]

But the 1.12.0 release doesn't have the file /tools/cythonize.py

So I tried again with an old version of SciPy, I saw the PR scipy/scipy#19320 about fixing the Cython version in 1.11.3 and releasing a final 1.11.4 version before the 1.12.0 release, using that version in the recipe also gave the original error.

mriscoc

mriscoc commented on Mar 25, 2025

@mriscoc
mriscoc
on Mar 25, 2025
ContributorAuthor

Is there anything that I can do to fix this recipe? The same error is produced with current development branch.

mriscoc
mentioned this on Mar 25, 2025
AndreMiras

AndreMiras commented on Mar 25, 2025

@AndreMiras
AndreMiras
on Mar 25, 2025
Member

I took a stab at it last summer and it was highly painful, hopefully things got better with recent releases, I don't know.
But basically you can start writing a recipe, try to build, fix the build errors by updating the recipe, rebuild...iterate until you have them all fixed.
Then do the same for the runtime errors.
You could probably start with:

from pythonforandroid.recipe import MesonRecipe
class ScipyRecipe(MesonRecipe):
    ...

Then the workflow I'm using to rebuild is:

# clean the build between rebuilds (not sure it's currently the best approach
p4a clean_recipe_build scipy && p4a clean_dists
# build using the testapps
cd testapps/on_device_unit_tests/
python setup.py apk \
  --sdk-dir $ANDROID_SDK_HOME \
  --ndk-dir $ANDROID_NDK_HOME \
  --arch=x86_64 \
  --requirements python3,scipy

In fact I often build from the repository docker image, this is the way I like the best, but you can also build from your host.
In case you want to give it a try from the container, this is how:

# build the image (or use the already published one)
docker build --tag=kivy/python-for-android .
# Docker run (note the volume mount so you don't have to rebuild the image after updating the recipe):
docker run -it \
  --volume $(pwd)/pythonforandroid:/home/user/app/pythonforandroid \
  --env ANDROID_NDK_HOME_LEGACY=/home/user/.android/android-ndk-legacy \
  --env ANDROID_SDK_HOME=/home/user/.android/android-sdk \
  --env ANDROID_NDK_HOME=/home/user/.android/android-ndk \
  --rm kivy/python-for-android bash
# then from within the container, same thing as above
. venv/bin/activate
cd testapps/on_device_unit_tests/
python setup.py apk --debug \
  --sdk-dir $ANDROID_SDK_HOME \
  --ndk-dir $ANDROID_NDK_HOME \
  --requirements python3,scipy \
  --arch=armeabi-v7a # or whatever arch you like

Good luck! 🏃

T-Dynamos

T-Dynamos commented on Mar 27, 2025

@T-Dynamos
T-Dynamos
on Mar 27, 2025
Contributor

Ok I went through all the pain, here you go: https://github.com/T-Dynamos/python-for-android/tree/scipy_update

You need to set ndk version to r27c (in spec) and ndk api to 24 (minapi).
Only works for 64 bit arches.

Here is my test:

Image

Apk size was 53MB

Test code:

import numpy as np
from scipy.integrate import quad

def f(x):
    return np.sin(x)

result, error = quad(f, 0, np.pi)
print(f"Result of the integral: {result}")
print(f"Estimated error: {error}")

Will open PR soon when will get more time (needs cleaning).

T-Dynamos
linked a pull request that will close this issue on Mar 27, 2025
mriscoc

mriscoc commented on Mar 27, 2025

@mriscoc
mriscoc
on Mar 27, 2025
ContributorAuthor

Oh thanks, that was quick! I just managed to build the configuration.

T-Dynamos

T-Dynamos commented on Mar 27, 2025

@T-Dynamos
T-Dynamos
on Mar 27, 2025 · edited by T-Dynamos
Contributor

But unfortunately, updating ndk to 27c doesn't allows SDL to build.

I had to pull trick for that:

1. build with only python3 and kivy (in requirements) with ndk 25b
2. change ndk to 27c with scipy and numpy in requirements

then it should work.

Edit: NVM fixed in latest commit.

mriscoc

mriscoc commented on Mar 27, 2025

@mriscoc
mriscoc
on Mar 27, 2025 · edited by mriscoc
ContributorAuthor

@T-Dynamos

Edit: NVM fixed in latest commit.

I'm getting this error with your fix:

[INFO]:    Downloading sdl2
[INFO]:    -> directory context /home/mrisco/kivy/test_scipy/.buildozer/android/platform/build-x86_64/packages/sdl2
[DEBUG]:   -> running basename https://github.com/libsdl-org/SDL/releases/download/release-2.30.11/SDL2-2.30.11.tar.gz
[DEBUG]:        SDL2-2.30.11.tar.gz
[DEBUG]:   * Generated md5sum: bea190b480f6df249db29eb3bacfe41e
[DEBUG]:   * Expected md5sum: a344eb827a03045c9b399e99af4af13d
...
ValueError: Generated md5sum does not match expected md5sum for sdl2 recipe

By changing the line 11 of pythonforandroid/recipes/sdl2/init.py to

md5sum = 'bea190b480f6df249db29eb3bacfe41e'

fix that.

But I got a new error:

[INFO]:    Prebuilding scipy for x86_64
[INFO]:    scipy has no prebuild_x86_64, skipping
[INFO]:    Applying patches for scipy[x86_64]
[INFO]:    Applying patch meson.patch
[DEBUG]:   -> running patch -t -d /home/mrisco/kivy/test_scipy/.buildozer/android/platform/build-x86_64/build/other_builds/scipy/x86_64__ndk_target_24/scipy -p1 -i /home/mrisco/python/python-for-android/pythonforandroid/recipes/scipy/meson.patch
[DEBUG]:        (Stripping trailing CRs from patch; use --binary to disable.)
[DEBUG]:        can't find file to patch at input line 5
[DEBUG]:        Perhaps you used the wrong -p or --strip option?
[DEBUG]:        The text leading up to this was:
[DEBUG]:        --------------------------
[DEBUG]:        |Binary files scipy.git/.git/index and scipy.git.patch/.git/index differ
[DEBUG]:        |diff '--color=auto' -uNr scipy.git/.git/logs/refs/remotes/origin/main scipy.git.patch/.git/logs/refs/remotes/origin/main
[DEBUG]:        |--- scipy.git/.git/logs/refs/remotes/origin/main       2025-03-27 02:55:14.521123150 +0530
[DEBUG]:        |+++ scipy.git.patch/.git/logs/refs/remotes/origin/main 2025-03-27 11:24:34.225186085 +0530
[DEBUG]:        --------------------------
[DEBUG]:        No file to patch.  Skipping patch.

That can be fixed by removing the references to .git folder in the meson.patch file:

diff '--color=auto' -uNr scipy.git/meson.options scipy.git.patch/meson.options
--- scipy.git/meson.options	2025-03-27 02:55:14.586853766 +0530
+++ scipy.git.patch/meson.options	2025-03-27 02:07:29.736674085 +0530
@@ -2,6 +2,8 @@
         description: 'option for BLAS library switching')
 option('lapack', type: 'string', value: 'openblas',
         description: 'option for LAPACK library switching')
+option('openblas_incldir', type: 'string', value: '', description: 'OpenBLAS include directory')
+option('openblas_libdir', type: 'string', value: '', description: 'OpenBLAS library directory')
 option('use-g77-abi', type: 'boolean', value: false,
         description: 'If set to true, forces using g77 compatibility wrappers ' +
                      'for LAPACK functions. The default is to use gfortran ' +
diff '--color=auto' -uNr scipy.git/scipy/meson.build scipy.git.patch/scipy/meson.build
--- scipy.git/scipy/meson.build	2025-03-27 02:55:14.632428649 +0530
+++ scipy.git.patch/scipy/meson.build	2025-03-27 11:25:33.756445056 +0530
@@ -268,10 +268,18 @@
   endif
 endif

+openblas_inc = get_option('openblas_incldir')
+openblas_lib = get_option('openblas_libdir')
+
+openblas_dep = declare_dependency(
+  include_directories: include_directories(openblas_inc),
+  link_args: ['-L' + openblas_lib, '-lopenblas']
+)
+
 # pkg-config uses a lower-case name while CMake uses a capitalized name, so try
 # that too to make the fallback detection with CMake work
 if blas_name == 'openblas'
-  blas = dependency(['openblas', 'OpenBLAS'])
+  blas = openblas_dep
 elif blas_name != 'scipy-openblas'  # if so, we found it already
   blas = dependency(blas_name)
 endif
@@ -295,7 +303,7 @@
   # use that - no need to run the full detection twice.
   lapack = blas
 elif lapack_name == 'openblas'
-  lapack = dependency(['openblas', 'OpenBLAS'])
+  lapack = openblas_dep
 else
   lapack = dependency(lapack_name)
 endif
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    recipe

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      Participants

      @AndreMiras@mriscoc@kuzeyron@T-Dynamos

      Issue actions
        SciPy recipe fails with current development branch of P4A · Issue #3116 · kivy/python-for-android