Scipy

[freakboy3742]

What is the PyPI name of the package you would like to see added?

scipy

Additional details

Major blocker is availability of a Fortran compiler that can target iOS

[freakboy3742]

Via @ydesgagn - he reports that it's possible to get flang working for iOS by running it through a Docker container.

Use this docker file to prepare a flang image:

FROM ubuntu:jammy

# persistent dependencies
RUN set -eux ; \
    apt-get -y update ; \
    apt-get -y --no-install-recommends install ca-certificates build-essential wget libncurses5 ;

# flang
RUN wget https://github.com/flang-compiler/flang/releases/download/flang_20190329/flang-20190329-x86-70.tgz ; \
    tar -xvzf *.tgz ; \
    rm -f *.tgz ; \
    ldconfig ;

Then, use this script (flang.sh) to execute the container:

#!/usr/bin/env bash

# shellcheck disable=SC2068
docker exec -it flang /root/host/proxy.sh "${PWD}" ${@}
cp -rf inbox/* "${PWD}" &> /dev/null
rm -rf inbox/* &> /dev/null

The replacement gfortran is a Python script that fixes command line arguments and calls flang.sh:

#!/usr/bin/env python3

import sys
import os
import shlex
import subprocess
import string
import random
import shutil

def join_args(args):
    return ' '.join(shlex.quote(x) for x in args)

if "-dumpversion" in sys.argv:
    print("10.2.0")
    sys.exit(0)

arguments = []
sys_args = sys.argv
del sys_args[0]

try:
    for arg in os.environ["ARCHFLAGS"].split(" "):
        sys_args.append(arg)
except KeyError:
    pass

if "-h" in sys_args or "--help" in sys_args:
    print("The running executable emulates a Fortran compiler. If '-arch arm64' is passed, the compiler will produce an arm64 object file for iOS from the passed source. If not, gfortran located in '/opt/homebrew/bin/' will be executed.")
    print("Because flang runs in a Docker container, only files under '/Users/', '/var/folders' or '/Library' can be compiled.")
    sys.exit(0)

all_args_are_object_files = True

for arg in sys_args:

    if os.path.isfile(arg) and not arg.endswith(".o"):
        all_args_are_object_files = False

if (not "-c" and not all_args_are_object_files) in sys_args or not "-arch arm64" in " ".join(sys_args):
    print("The executed Fortran compiler only supports producing object files for iOS arm64, falling back to gfortran.", file=sys.stderr)
    print("To compile sources for iOS arm64, make sure to add -arch arm64 and -c.", file=sys.stderr)
    sys.exit(os.system(shlex.join(["/opt/homebrew/bin/gfortran"]+sys_args)))

if "-bundle" in sys_args or all_args_are_object_files:
    args = ["clang", "-undefined", "dynamic_lookup", "-shared"]
    try:
        for arg in os.environ["LDFLAGS"].split(" "):
            args.append(arg)
    except KeyError:
        pass
    for arg in sys_args:
        if arg != "-bundle":
            args.append(arg)
    
    command = shlex.join(args)
    sys.exit(os.system(command))

def convert_to_docker_path(arg):
    if arg.startswith("/"):
        arg = "/¡"+arg
    else:
        arg = os.path.join(os.getcwd(), arg)
    
    return arg

the_previous_parameter_was_dash_o = False
the_previous_parameter_was_dash_c = False
the_previous_parameter_was_dash_arch = False
output_path = None
source = None

for arg in sys_args:

    if arg == "-c":
        the_previous_parameter_was_dash_c = True
    elif the_previous_parameter_was_dash_c:
        the_previous_parameter_was_dash_c = False
        source = arg

    if arg == "-o":
        the_previous_parameter_was_dash_o = True
        continue
    elif the_previous_parameter_was_dash_o:
        the_previous_parameter_was_dash_o = False
        output_path = arg
        continue

    if arg == "-arch":
        the_previous_parameter_was_dash_arch = True
        continue
    elif the_previous_parameter_was_dash_arch:
        the_previous_parameter_was_dash_arch = False
        continue

    if arg == "-fallow-argument-mismatch":
        continue

    if os.path.exists(arg):
        arg = convert_to_docker_path(arg)
    if arg.startswith("-I"):
        path = arg.split("-I")[-1]
        arg = "-I"+convert_to_docker_path(path)

    if arg.startswith("/¡"):
        arg = arg[2:]

    arguments.append(arg)

if output_path is None and source is not None:
    parts = source.split(".")
    del parts[-1]
    output_path = os.getcwd()+"/"+".".join(parts)+".o"

if os.path.isfile(output_path):
    sys.exit(0)

print(output_path)

dir = os.path.dirname(os.path.abspath(__file__))
cwd = os.getcwd()

arguments.insert(0, os.path.abspath(os.path.join(dir, "flang.sh")))
arguments.insert(1, "--save-temps")
flang_command = join_args(arguments)

inbox = os.path.join(cwd, ".inbox"+''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(5)))

try:
    os.mkdir(inbox)
except FileExistsError:
    pass

os.chdir(inbox)

os.system(flang_command)
file_path = None

for file in os.listdir("."):
    if not file.endswith(".ll"):
        try:
            os.remove(file)
        except FileNotFoundError:
            pass
    else:
        file_path = os.path.join(os.getcwd(), file)

os.chdir(cwd)
llc = [os.path.join(dir, "llc"), "-mtriple=arm64-apple-ios", "-filetype=obj", file_path, "-o", output_path]
subprocess.run(llc, stdout=None, stderr=None)

shutil.rmtree(inbox)

The proxy.sh that is referenced is:

#!/usr/bin/env bash

cd /root/host || exit
cd "${1}" || exit
flang "${@:2}"

[mhsmith]

Major blocker is availability of a Fortran compiler that can target iOS

@meow464: You used to have a good summary of this at https://meow464.neocities.org/Fortran/state-of-fortran-on-ios. Is there some reason why you've taken it down?

[meow464]

The url changed https://meow464.neocities.org/blog/state-of-fortran-on-ios/

I added support for iOS cross compilation on LFortran but it still can't compile scipy and lapack. Both are being actively worked on.

lfortran/lfortran#1251

[meow464]

I will add instructions for cross compiling for iOS, there was a pull request for that but I forgot about it.

It's not complicated, just cross compile the runtime with a iOS cmake toolchain file then use --target=apple-iphone-ios

[meow464]

@freakboy3742 is that the old or new flang?

[freakboy3742]

@meow464 I can't say I know - I opened this ticket so I had a store of some details that @ydesgagn provided to me; he may be able to comment. If I had to guess based on the Dockerfile, it seems to be using flang 20190329.

[meow464]

That's the old flang also known as flang classic, it's based on the open sourced PGI compiler and it has a problem generating the apple-arm ABI. I spent some 6 months working on it in 2021 but couldn't fix it.
Yes it can generate code and some stuff will work but other stuff will crash.

The problem is with variadic arguments. On arm they should be put on registers until none are available and then on the stack. On apple they should all be put on the stack.

The flang intrinsic functions are written in C and compiled with the apple abi (as they should) but the fortran code is compiled with the arm abi.

[freakboy3742]

Ah - good to know. Thanks for those details. "Run flang through Docker" seemed like a neat trick, but it also seemed like just enough of a neat trick that there was very likely a land mine in waiting.

[freakboy3742]

It's worth noting that Quansight recently blogged about getting flang working with SciPy.

[ydesgagn]

@meow464 We are executing unit tests on an iPad M2 and they all pass for Scipy. Can you be more specific on has a problem generating the apple-arm ABI? Also we have an iPad only application using written in Swift accepted by the Apple Store extensively calling Python 3.11 with numpy, scikit-learn and scipy and all the reported bugs we got so far are due to our failure in our code. Nothing in Python, numpy, scikit-learn and scipy.

@freakboy3742 one of my guy is about to PR in https://github.com/beeware/mobile-forge to add scikit-learn. Will do more after that and we can try Scipy if you think the Docker way is not too hacky.

[freakboy3742]

@ydesgagn "Working with a weird hack" is better than "not working"... but I guess it depends how much engineering overhead we'll need to add to support the hack. If it's something that we can add now, then remove easily later if/when a "direct" flang approach works, then I don't mind living with a hack.

Suffice to say, I'd much rather see a "direct" flang approach - but I'd also like a pony, and a gold Lamborghini :-)

[mhsmith]

Can you be more specific on has a problem generating the apple-arm ABI?

There are some more details and links here, but they may not be up to date.

[serhii-vlasiuk]

For 3.11
Added recipes/scipy/meta.yaml
package:
name: scipy
version: 1.11.3

Getting error
ERROR: Could not find a version that satisfies the requirement numpy==1.23.2 (from versions: 1.26.0)
ERROR: No matching distribution found for numpy==1.23.2
Traceback (most recent call last):
File "/Users/svlas/ydesgagn/github/mobile-forge/venv3.11/bin/forge", line 8, in
sys.exit(main())
^^^^^^
File "/Users/svlas/ydesgagn/github/mobile-forge/src/forge/main.py", line 205, in main
builder.prepare(clean=args.clean and (p == 0))
File "/Users/svlas/ydesgagn/github/mobile-forge/src/forge/build.py", line 407, in prepare
self.cross_venv.pip_install(
File "/Users/svlas/ydesgagn/github/mobile-forge/src/forge/cross.py", line 375, in pip_install
self.run(
File "/Users/svlas/ydesgagn/github/mobile-forge/src/forge/cross.py", line 362, in run
return subprocess.run(args, **final_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/svlas/ydesgagn/github/Python-Apple-support/install/macOS/macosx/python-3.11.6/lib/python3.11/subprocess.py", line 575, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python', '-m', 'pip', 'install', '--disable-pip-version-check', '--only-binary', ':all:', '--find-links', '/Users/svlas/ydesgagn/github/mobile-forge/dist', 'build', 'meson-python>=0.12.1,<0.15.0', 'Cython>=0.29.35,<3.0', 'pybind11>=2.10.4,<2.11.1', 'pythran>=0.12.0,<0.15.0', "numpy==1.22.0; platform_machine=='loongarch64'", "numpy==1.22.3; python_version=='3.9' and platform_system=='Windows' and platform_python_implementation != 'PyPy'", "numpy==1.22.3; python_version=='3.10' and platform_system=='Windows' and platform_python_implementation != 'PyPy'", "numpy==1.21.6; python_version=='3.9' and (platform_system!='Windows' and platform_machine!='loongarch64') and platform_python_implementation != 'PyPy'", "numpy==1.21.6; python_version=='3.10' and (platform_system!='Windows' and platform_machine!='loongarch64') and platform_python_implementation != 'PyPy'", "numpy==1.23.2; python_version=='3.11' and platform_python_implementation != 'PyPy'", "numpy>=1.26.0,<1.27.0; python_version=='3.12'", "numpy; python_version>='3.9' and platform_python_implementation=='PyPy'", "numpy>=1.26.0; python_version>='3.13'"]' returned non-zero exit status 1.

Before built numpy 1.26.0
mobile-forge/dist/numpy-1.26.0-cp311-cp311-ios_12_0_iphoneos_arm64.whl

Will appreciate any advice?

[freakboy3742]

It seems to me that the error message is telling you exactly what is going on:

No matching distribution found for numpy==1.23.2

You need to have numpy 1.23.2 available for installation.

On 3.11, 1.23.2 is a special version of numpy - it's the oldest supported version. A lot of numpy-related tooling will deliberately build against the "oldest-supported-numpy" target to ensure maximum ABI compatibility. You'll need to build this version first (forge iOS numpy:1.23.2)

[h-vetinari]

You can patch out the dependence on oldest-supported-numpy and just use the oldest one you have available. It's just a question of being conservative with ABI support, it's not like that exact version is needed to build

[freakboy3742]

Sure, you can patch it out, but that's not how official Scipy builds are made. The goal of this project is to provide a stepping stone towards Scipy providing its own official iOS packages (although, to be clear - that's a way off). If we're going to patch SciPy, the patches are going to be specific to getting a successful build on iOS, not hacks to get around inconveniences that have no chance of being upstreamed.

Plus - it's not like it's hard to build numpy 1.23.2. The recipe is already there.

[hyazel]

It's super cool that you're working on it, thanks.

I was just struggling making scipy work on an iOS project.

Meanwhile I'll take a look on the mobile forge mechanism.

[hyazel]

I tried to forge iOS scipy with numpy (1.23.3) already compiled on my machine but I have the following error :

numpy.distutils.system_info.NotFoundError: No BLAS/LAPACK libraries found.
To build Scipy from sources, BLAS & LAPACK libraries need to be installed.

seems to be related to the Fortran compilation you mentionned earlier.

Do you have an ongoing PR @freakboy3742 or something we can help you on ?

@ydesgagn Hello ! Could you describe the all process to compile scipy on Docker ?

Thanks !

[ydesgagn]

@hyazel will create a PR this weekend so you can try with the proper tools available.

[freakboy3742]

Do you have an ongoing PR @freakboy3742 or something we can help you on ?

Everything I have in progress for mobile forge is open on PRs on this repo at present. I haven't tackled BLAS/LAPACK at all.

[pasdeloup]

Hello, I'm interested in the topic too.
I worked with @mhsmith on making SciPy 1.8.1 work on Android with Chaquopy and now my team needs the same version working on iOS too to compile our prototype.
@ydesgagn I couldn't find the PR you mentioned, can I find somewhere those proper tools? As it's seem to be Docked based, can I cross-compile from a Linux?
Thanks.

[freakboy3742]

@pasdeloup We'd definitely appreciate any assistance you can provide. SciPy is a big obvious gap in BeeWare's iOS offering, which we'd like to address.

All the details that I have are in this ticket; I don't know if that gives you anything to work with, but I'm happy to provide guidance where I can.

[ydesgagn]

Here the PR #28

Really sorry for not responding sooner. I'm no longer familiar with the code the repo but if somebody can add the recipe with the instructions in the PR, this was working for me in the past with:

From Python-Apple-support:

Python version: 3.11.3
Build: custom
Min iOS version: 12.0
libFFI: 3.4.2
BZip2: 1.0.8
OpenSSL: 3.1.0
XZ: 5.4.2

Then I build the following with this hack:

numpy: 1.24.2
scikit-learn: 1.2.2
scipy: 1.11.1

I will try to help if you run out in any trouble.

[freakboy3742]

@ydesgagn Thanks for that PR. I'm not sure when I'll get a chance to take a deep look at it myself, but if anyone else wants to dig into the problem, they've now got all the pieces.

[pasdeloup]

Thanks @ydesgagn
A teammate is already working on it (I'm on Linux not Mac), we were trying to figure out what you did from your other repo
https://github.com/Cloud-Officer/ios-frameworks but this PR will help a lot!

So you don't need to build first BLAS/LAPACK?

[ydesgagn]

No I did not built it. I just pointed scipy at the homebrew ARM64 version and linked with that. But I know @freakboy3742 is not a big fan of homebrew dependencies.

[freakboy3742]

I'm not a fan because homebrew libraries have a history of breaking iOS builds because they're linked against macOS system libraries, not iOS system libraries (gettext is a particularly recurring culprit here)

If the homebrew build of BLAS/LAPACK does work on iOS, then it should be possible to port the recipe directly to mobile-forge, except explicitly using iOS compilers and libraries, rather than relying on a lack of problematic symbols. We already do builds for libraries like libpng and libjpeg; if we've got a working recipe, I can't see why BLAS/LAPACK would be any different.

[mhsmith]

It looks like there are already official BLAS and LAPACK builds from Apple (see OS support indicators at the top). Maybe we could use these, although SciPy might need a bit of help to find them.

[noelmullankuzhy]

@pasdeloup Hi, do you have any progress on building SciPy wheel for iOS?. @ydesgagn Could you please share the working SciPy wheel that you have built, while we figure out how to make build on out own? Thank you.

[michaeleisel]

SciPy works well with Apple's BLAS/LAPACK library, Accelerate. It can be built from the SciPy repo with python dev.py build --with-accelerate. This strategy is also what allows NumPy to work. However, SciPy also has its own Fortran files outside of BLAS/LAPACK, such as in scipy/integrate/quadpack. This is where we still need a Fortran compiler.

It appears that the Pyto iOS app supports SciPy, and that app's author has found a way to compile Fortran code with Docker, similar to the approach discussed above. Their repo: https://github.com/ColdGrub1384/fortran-ios . Presumably, this is the tool they're using for SciPy for Pyto.

@ColdGrub1384 are you able to share a more fleshed-out list of steps for compiling these Fortran files for SciPy? For example, how did you deal with things like, the linking in of /opt/homebrew/Cellar/gcc/14.1.0_2/lib/gcc/current to the Fortran .so files, did you cross-compile that library as well?

Another strategy would be to take the .so files for macOS and just change their binary metadata so that they look like iOS binaries, and then use them like that. This is a little weird, but I'm told that it's pretty safe.

[meow464]

Scipy used to support Accelerate but no longer does, the reasons are listed here: https://github.com/scipy/archive/blob/main/wiki/dropping-accelerate-support.md

[freakboy3742]

@meow464 Are you sure that information is current? I know it was definitely true in the SciPy v1.2 timeframe (which is where that document comes from), but conversations I've had recently suggest that that the SciPy team either have moved, or are planning to move back to Accelerate because the bugs in the Accelerate implementation of BLAS/LAPACK have become less of a problem than the general problem of trying to get a Fortran compiler that works.

[meow464]

I do not know if it's current and good to hear they are considering Accelerate again.

[h-vetinari]

Since apple introduced an updated LAPACK in MacOS 13.3, we have readded explicit support, see scipy/scipy#19816