f2py entry point with non-functioning shebang in osx-arm cross-compile #276

bnavigator · 2022-07-11T13:46:21Z

Solution to issue cannot be found in the documentation.

I checked the documentation.

Issue

With

requirements:
  build: 
    - numpy                   # [build_platform != target_platform]

for the osx-arm cross-compilation, calling f2py3 fails with:

/Users/runner/miniforge3/conda-bld/slycot_1657493761007/_build_env/bin/f2py3: line 3: import: command not found
/Users/runner/miniforge3/conda-bld/slycot_1657493761007/_build_env/bin/f2py3: line 4: import: command not found
from: can't read /var/mail/numpy.f2py.f2py2e
/Users/runner/miniforge3/conda-bld/slycot_1657493761007/_build_env/bin/f2py3: line 7: syntax error near unexpected token `('
/Users/runner/miniforge3/conda-bld/slycot_1657493761007/_build_env/bin/f2py3: line 7: `    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])'

$> which f2py3
$BUILD_PREFIX/bin/f2py3
$> head f2py3
#!$BUILD_PREFIX/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from numpy.f2py.f2py2e import main

See discussion in conda-forge/slycot-feedstock#50 and conda-forge/slycot-feedstock#49.

cc @conda-forge/help-osx-arm64

Installed packages

bzip2:           1.0.8-h3422bc3_4              conda-forge
    c-ares:          1.18.1-h3422bc3_0             conda-forge
    ca-certificates: 2022.6.15-h4653dfc_0          conda-forge
    cmake:           3.23.2-h52850c7_0             conda-forge
    distro:          1.6.0-pyhd8ed1ab_0            conda-forge
    expat:           2.4.8-h6b3803e_0              conda-forge
    krb5:            1.19.3-he492e65_0             conda-forge
    libblas:         3.9.0-5_h880f123_netlib       conda-forge
    libcblas:        3.9.0-5_h880f123_netlib       conda-forge
    libcurl:         7.83.1-h7965298_0             conda-forge
    libcxx:          14.0.6-h04bba0f_0             conda-forge
    libedit:         3.1.20191231-hc8eb9b7_2       conda-forge
    libev:           4.33-h642e427_1               conda-forge
    libffi:          3.4.2-h3422bc3_5              conda-forge
    libgfortran:     5.0.0.dev0-11_0_1_hf114ba7_23 conda-forge
    libgfortran5:    11.0.1.dev0-hf114ba7_23       conda-forge
    liblapack:       3.9.0-5_h880f123_netlib       conda-forge
    libnghttp2:      1.47.0-hf30690b_0             conda-forge
    libssh2:         1.10.0-h7a5bd25_2             conda-forge
    libuv:           1.43.0-h3422bc3_0             conda-forge
    libzlib:         1.2.12-ha287fd2_1             conda-forge
    llvm-openmp:     14.0.4-hd125106_0             conda-forge
    lz4-c:           1.9.3-hbdafb3b_1              conda-forge
    make:            4.3-he57ea6c_1                conda-forge
    ncurses:         6.3-h07bb92c_1                conda-forge
    numpy:           1.23.1-py310h0a343b5_0        conda-forge
    openssl:         3.0.5-ha287fd2_0              conda-forge
    packaging:       21.3-pyhd8ed1ab_0             conda-forge
    pip:             22.1.2-pyhd8ed1ab_0           conda-forge
    pyparsing:       3.0.9-pyhd8ed1ab_0            conda-forge
    python:          3.10.5-h4eee789_0_cpython     conda-forge
    python_abi:      3.10-2_cp310                  conda-forge
    readline:        8.1.2-h46ed386_0              conda-forge
    rhash:           1.4.3-he4db4b2_0              conda-forge
    scikit-build:    0.15.0-pyhb871ab6_0           conda-forge
    setuptools:      63.1.0-py310hbe9552e_0        conda-forge
    sqlite:          3.39.0-h40dfcc0_0             conda-forge
    tk:              8.6.12-he1e0b03_0             conda-forge
    tzdata:          2022a-h191b570_0              conda-forge
    wheel:           0.37.1-pyhd8ed1ab_0           conda-forge
    xz:              5.2.5-h642e427_1              conda-forge
    zlib:            1.2.12-ha287fd2_1             conda-forge
    zstd:            1.5.2-hd705a24_2              conda-forge


    bzip2:                       1.0.8-h0d85af4_4          conda-forge
    c-ares:                      1.18.1-h0d85af4_0         conda-forge
    ca-certificates:             2022.6.15-h033912b_0      conda-forge
    cctools_osx-64:              973.0.1-h3eff9a4_10       conda-forge
    cctools_osx-arm64:           973.0.1-h98580c8_10       conda-forge
    clang:                       13.0.1-h694c41f_0         conda-forge
    clang-13:                    13.0.1-default_he082bbe_0 conda-forge
    clang_osx-arm64:             13.0.1-hdd0e76e_2         conda-forge
    clangxx:                     13.0.1-default_he082bbe_0 conda-forge
    cmake:                       3.23.2-hf2c7296_0         conda-forge
    compiler-rt:                 13.0.1-he01351e_0         conda-forge
    compiler-rt_osx-64:          13.0.1-hd3f61c9_0         conda-forge
    cross-python_osx-arm64:      3.10-27_cpython           conda-forge
    crossenv:                    1.2.0-pyhd8ed1ab_7        conda-forge
    distro:                      1.6.0-pyhd8ed1ab_0        conda-forge
    expat:                       2.4.8-h96cf925_0          conda-forge
    gettext:                     0.19.8.1-hd1a6beb_1008    conda-forge
    gfortran_impl_osx-64:        9.3.0-h9cc0e5e_23         conda-forge
    gfortran_impl_osx-arm64:     11.0.1.dev0-h3653101_23   conda-forge
    gfortran_osx-arm64:          11.0.1.dev0-h64a2375_15   conda-forge
    gmp:                         6.2.1-h2e338ed_0          conda-forge
    isl:                         0.22.1-hb1e8313_2         conda-forge
    krb5:                        1.19.3-hb98e516_0         conda-forge
    ld64_osx-64:                 609-h6fbe7a8_10           conda-forge

Environment info

(conda-forge osx-arm build on azure)

The text was updated successfully, but these errors were encountered:

h-vetinari · 2022-07-11T16:09:29Z

Thanks for the report @bnavigator. Seems there's a bunch more relevant info in conda-forge/slycot-feedstock#50.

@isuruf @beckermr @xhochy
How are entry-points handled/supported in cross-compilation (if at all)? Seems like the current f2py entry point doesn't like it to be called while cross-compiling for osx-arm.

beckermr · 2022-07-11T16:31:42Z

@erykoff may have insight here too

bnavigator · 2022-07-11T16:37:54Z

The explanation in conda-forge/slycot-feedstock#50 (comment) doesn't look plausible to me.

beckermr · 2022-07-11T16:45:33Z

Did you try the suggested change? conda-build won't know about entry points unless you tell it about them. It may be doing special handing for them for cross-compiling (and does for some cases like noarch builds already).

h-vetinari · 2022-07-11T16:49:29Z

I overlooked the # [win] selector when checking that the entry-point is defined, apologies.

bnavigator · 2022-07-11T16:49:43Z

How am I supposed to change the numpy-feedstock for tests in my own feedstock?
There is nothing in the documentation about regular entry_points (unless for noarch packages). Since when is there a requirement to duplicate the python package entry point specification in the recipe?

beckermr · 2022-07-11T16:54:08Z

You can build the numpy version/recipe you need locally and try cross-compiling the recipe against the local numpy package with the change.
I am not claiming there is a requirement but merely postulating that there might be one for cross-compiling. There may not be. I don't know.

As it seems easy to test this out, it seems like a worthwhile step.

FYI, both Isuru and Uwe are either out of contact completely or rarely in contact for at least a few months so you may not get help from them right away. They both know more than I do about this so we may have to wait to get this resolved.

bnavigator · 2022-07-11T16:55:01Z

Negative. I don't have a mac to test this locally.

bnavigator · 2022-07-11T16:56:14Z

Open unanswered question: conda/conda-build#4291

Knowledgebase about noarch entry_points, unclear about what is needed on non-noarch: https://conda-forge.org/docs/maintainer/knowledge_base.html#noarch-python

beckermr · 2022-07-11T16:56:17Z

Ahhh then we'll have to wait until someone who does have a mac has the time or someone more knowledgeable can confirm what change we need. :/

h-vetinari · 2022-07-11T16:57:54Z

What's the harm in building this feedstock without the # [win] selector for the entry-point? The tests contain the CLI call to f2py.

And even if we don't feel comfortable in doing that, we could push the builds to a dev label, for example.

beckermr · 2022-07-11T16:59:25Z

Good find on that conda-build issue! Many of the core conda-build contributors are not active on the project anymore for various reasons and so that question might best be answered by grepping the code.

@h-vetinari Great idea! Using a dev label is totally fine. If you need me to make a branch in the upstream feedstock just bump in the PR to be merged to it.

bnavigator · 2022-07-11T16:59:26Z

What's the harm in building this feedstock without the # [win] selector for the entry-point? The tests contain the CLI call to f2py.

You would have to make an additional line for nowin (or whatever is the correct conditional) as the # [win] line adds a specific batch wrapper.

beckermr · 2022-07-11T17:00:23Z

You would have to make an additional line for nowin (or whatever is the correct conditional) as the # [win] line adds a specific batch wrapper.

Seems not so hard to do with a little time and elbow grease. Feel free to go ahead if you'd like.

h-vetinari · 2022-07-11T17:02:42Z

@h-vetinari Great idea! Using a dev label is totally fine. If you need me to make a branch in the upstream feedstock just bump in the PR to be merged to it.

I'm a maintainer on this repo, so I can do that myself. 🙃

You would have to make an additional line for nowin (or whatever is the correct conditional) as the # [win] line adds a specific batch wrapper.

It's called # [not win] or # [unix]. If you an let me know the correct path then I have a PR almost ready already.

erykoff · 2022-07-11T17:04:37Z

That is a great question on the conda-build issue! And I may be wrong that this is the source of the problem, and I certainly don't know the interaction of the entry points defined in setup and those in meta.yaml, and how this works with the cross-python ... but it certainly is suspicious, and I've had problems with incorrect entry points in the meta.yaml messing things up (but maybe these were noarch packages which might matter?).

In an admittedly quick search, I couldn't find any prior art in using f2py in particular in a cross-compiling setting.

bnavigator · 2022-07-11T17:06:47Z

Hmm, looking at it a little longer, doesn't the entry_points line contradict the batch file?

numpy-feedstock/recipe/meta.yaml

Line 25 in a823229

- f2py = numpy.f2py.f2py2e:main # [win]

numpy-feedstock/recipe/f2py.bat

Lines 1 to 4 in a823229

    
           :: This wrapper script is necessary to make the "f2py" command on windows work 
        
           @SET "PYTHON_EXE=%~dp0\..\python.exe" 
        
           call "%PYTHON_EXE%" "%~dp0\f2py.py" %*

numpy-feedstock/recipe/bld.bat

Lines 19 to 22 in a823229

    
           XCOPY %RECIPE_DIR%\f2py.bat %SCRIPTS% /s /e 
        
           if errorlevel 1 exit 1 
        
           del %SCRIPTS%\f2py.exe

So maybe, the correct entries should actually be # [no win] and be present for f2py3 and f2py3.10 as well:

https://github.com/numpy/numpy/blob/3da5da966d4518f7312bda4872b51f9f21de50a9/setup.py#L419-L429
https://github.com/numpy/numpy/blob/3da5da966d4518f7312bda4872b51f9f21de50a9/setup.py#L455

bnavigator · 2022-07-11T17:09:22Z

In an admittedly quick search, I couldn't find any prior art in using f2py in particular in a cross-compiling setting.

There is as least https://github.com/conda-forge/fastscapelib-f2py-feedstock, but they also patch the F2PY_EXECUTABLE from the CMakeLists.txt away. (conda-forge/fastscapelib-f2py-feedstock#17)

h-vetinari · 2022-07-11T17:15:29Z

So maybe, the correct entries should actually be # [no win] and be present for f2py3 and f2py3.10 as well:

I did the equivalent in #277. I'm not convinced we still need f2py.bat at all if the current python-based entry-point is working. It's only lightly tested though f2py -h, so we could consider expanding that. I'm divided (though not very passionately) about adding f2py3 but against adding f2py3.10.

bnavigator · 2022-07-11T17:16:36Z

f2py3 and f2py3.10 are upstream definitions and expected e.g. by https://github.com/scikit-build/scikit-build/blob/master/skbuild/resources/cmake/FindF2PY.cmake#L69

Currently they are being installed (by build.sh -> pip), but not registered by conda-build.

bnavigator · 2022-07-11T17:19:27Z

I'm not convinced we still need f2py.bat at all if the current python-based entry-point is working

Did you check whether the entry_points entry overrides the .bat wrapper copied in bld.bat or if it is the other way round (del f2py.exe)?

h-vetinari · 2022-07-11T17:28:54Z

I'm not convinced we still need f2py.bat at all if the current python-based entry-point is working

Did you check whether the entry_points entry overrides the .bat wrapper copied in bld.bat or if it is the other way round (del f2py.exe)?

That's an interesting question. f2py.bat was last touched at a time when no entry-point specification existed in meta.yaml (meaning it definitely used the bld.bat stuff). I don't know what takes precedence currently, since the entry-point specification was added. Both of the most "recent" changes in that regard were done by @ocefpaf, but I'd be surprised if he remembered the details. 😅

bnavigator · 2022-07-13T20:11:39Z

Not fixed by #277 :(
See conda-forge/slycot-feedstock#55

bnavigator added the bug label Jul 11, 2022

h-vetinari mentioned this issue Jul 11, 2022

Also define entry-point on unix #277

Merged

bnavigator mentioned this issue Jul 12, 2022

Warn if package contains console_scripts without listing entry_points conda/conda-build#3965

Closed

varlackc mentioned this issue Sep 1, 2022

Update Numpy to Version 1.23.2 AnacondaRecipes/numpy-feedstock#52

Closed

erykoff mentioned this issue Feb 3, 2023

ARM OSX Migrator conda-forge/openorb-feedstock#28

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

f2py entry point with non-functioning shebang in osx-arm cross-compile #276

f2py entry point with non-functioning shebang in osx-arm cross-compile #276

bnavigator commented Jul 11, 2022

h-vetinari commented Jul 11, 2022

beckermr commented Jul 11, 2022

bnavigator commented Jul 11, 2022

beckermr commented Jul 11, 2022

h-vetinari commented Jul 11, 2022

bnavigator commented Jul 11, 2022 •

edited

Loading

beckermr commented Jul 11, 2022

bnavigator commented Jul 11, 2022

bnavigator commented Jul 11, 2022 •

edited

Loading

beckermr commented Jul 11, 2022

h-vetinari commented Jul 11, 2022

beckermr commented Jul 11, 2022

bnavigator commented Jul 11, 2022

beckermr commented Jul 11, 2022

h-vetinari commented Jul 11, 2022

erykoff commented Jul 11, 2022

bnavigator commented Jul 11, 2022

bnavigator commented Jul 11, 2022

h-vetinari commented Jul 11, 2022 •

edited

Loading

bnavigator commented Jul 11, 2022 •

edited

Loading

bnavigator commented Jul 11, 2022

h-vetinari commented Jul 11, 2022 •

edited

Loading

bnavigator commented Jul 13, 2022

f2py entry point with non-functioning shebang in osx-arm cross-compile #276

f2py entry point with non-functioning shebang in osx-arm cross-compile #276

Comments

bnavigator commented Jul 11, 2022

Solution to issue cannot be found in the documentation.

Issue

Installed packages

Environment info

h-vetinari commented Jul 11, 2022

beckermr commented Jul 11, 2022

bnavigator commented Jul 11, 2022

beckermr commented Jul 11, 2022

h-vetinari commented Jul 11, 2022

bnavigator commented Jul 11, 2022 • edited Loading

beckermr commented Jul 11, 2022

bnavigator commented Jul 11, 2022

bnavigator commented Jul 11, 2022 • edited Loading

beckermr commented Jul 11, 2022

h-vetinari commented Jul 11, 2022

beckermr commented Jul 11, 2022

bnavigator commented Jul 11, 2022

beckermr commented Jul 11, 2022

h-vetinari commented Jul 11, 2022

erykoff commented Jul 11, 2022

bnavigator commented Jul 11, 2022

bnavigator commented Jul 11, 2022

h-vetinari commented Jul 11, 2022 • edited Loading

bnavigator commented Jul 11, 2022 • edited Loading

bnavigator commented Jul 11, 2022

h-vetinari commented Jul 11, 2022 • edited Loading

bnavigator commented Jul 13, 2022

bnavigator commented Jul 11, 2022 •

edited

Loading

bnavigator commented Jul 11, 2022 •

edited

Loading

h-vetinari commented Jul 11, 2022 •

edited

Loading

bnavigator commented Jul 11, 2022 •

edited

Loading

h-vetinari commented Jul 11, 2022 •

edited

Loading