CI: Prerelease, mypy, and freethreaded jobs failing due to Pythran error: `KeyError: 'sep'`

[nickodell]

Describe your issue.

The error KeyError: 'sep' can be seen in the prerelease, mypy, and freethreaded jobs.

Newest successful job I can find: https://github.com/scipy/scipy/actions/runs/19202532934/job/54892728452#step:9:313

Oldest failing job I can find: https://github.com/scipy/scipy/actions/runs/19208289070/job/54906718527#step:9:347

Differences and similarities between jobs

Both jobs use the following pip packages.

Successfully installed beniget-0.4.2.post1 click-8.2.1 cython-3.2.0 gast-0.6.0 importlib_metadata-8.7.0 meson-1.9.1 meson-python-0.18.0 ninja-1.13.0 numpy-2.3.4 packaging-25.0 ply-3.11 pybind11-3.0.1 pyproject-metadata-0.9.1 pythran-0.18.0 setuptools-80.9.0 spin-0.15 zipp-3.23.0
Successfully installed certifi-2025.10.5 charset_normalizer-3.4.4 contourpy-1.3.3 coverage-7.11.2 cycler-0.12.1 execnet-2.1.1 fonttools-4.60.1 gmpy2-2.3.0a1 hypothesis-6.147.0 idna-3.11 iniconfig-2.3.0 kiwisolver-1.4.10rc0 matplotlib-3.10.7 mpmath-1.4.0b2 pillow-12.0.0 platformdirs-4.5.0 pluggy-1.6.0 pooch-1.8.2 pygments-2.19.2 pyparsing-3.3.0a1 pytest-9.0.0 pytest-cov-7.0.0 pytest-xdist-3.8.0 python-dateutil-2.9.0.post0 requests-2.32.5 six-1.17.0 sortedcontainers-2.4.0 threadpoolctl-3.6.0 urllib3-2.5.0
Successfully installed scipy-openblas32-0.3.29.265.1
Successfully installed numpy-2.4.0.dev0

Though note that NumPy is installed from a nightly wheel - I am not sure it would show differences. Edit: this is why.

As far as I can tell, the two runs use the same apt packages and runner image as well.

Finding jobs

I have been using the following shell script to search for runs that have the error, or for runs that don't have the error but did compile the relevant file.

Usage is ./logdl.sh scipy/scipy <run id>

#!/usr/bin/env bash
set -euo pipefail

REPO_NAME=${1?Missing repo name argument} # example: vitejs/vite
RUN_ID=${2?Missing run ID argument}

ZIPFILE=$(mktemp)
TMPDIR=$(mktemp -d)

Color_Off='\033[0m'
BBlack='\033[1;30m'
On_Cyan='\033[46m'

function downloadLogs() {
  echo "downloading logs for run $RUN_ID ..."
  if ! gh api /repos/$REPO_NAME/actions/runs/$RUN_ID/logs > $ZIPFILE; then
    saw_failure=0
    meaningful_test=0
    echo "$RUN_ID $saw_failure $meaningful_test" | tee -a pr_parse.log
    exit 0
  fi
  unzip -q -d $TMPDIR $ZIPFILE
  rm -f $ZIPFILE
}

function dumpLogs() {
  find $TMPDIR -type f -maxdepth 1 -print0 |
    while IFS= read -r -d $'\0' file; do
      echo
      basefile=$(basename "$file")
      echo -e "${BBlack}${On_Cyan} $basefile ${Color_Off}"
      echo

      cat "$file"
    done
}

downloadLogs
echo "Downloaded to $TMPDIR"
set +e
(cd $TMPDIR; grep -C 5 "spec.kwonlydefaults" *Prerelease\ deps*.txt)
grep_kw_errorcode=$?
(cd $TMPDIR; grep -C 5 "Generating 'scipy/linalg/_linalg_pythran" *Prerelease\ deps*.txt)
grep_gen_errorcode=$?
set -e

if [ "$grep_kw_errorcode" -eq 0 ]; then
     saw_failure=1
     meaningful_test=1
fi
if [ "$grep_kw_errorcode" -eq 1 ]; then
     saw_failure=0
     # Did we actually reach the point where we could have seen a failure?
     if [ "$grep_gen_errorcode" -eq 0 ]; then
         meaningful_test=1
     else
         meaningful_test=0
     fi
fi
if [ "$grep_kw_errorcode" -eq 2 ]; then
     saw_failure=0
     meaningful_test=0
fi
echo "$RUN_ID $saw_failure $meaningful_test" | tee -a pr_parse.log

Reproducing Code Example

I seem to be able to reproduce this by installing NumPy nightly or a build from main. Numpy 2.3.4 does not exhibit the problem.

pip install --pre --upgrade --timeout=60 -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple numpy
# or
pip install git+https://github.com/numpy/numpy.git

Other dependency versions:

array_api_strict==2.4.1
asv==0.6.5
asv_runner==0.2.1
attrs==25.4.0
beniget==0.4.2.post1
build==1.3.0
certifi==2025.10.5
charset-normalizer==3.4.4
click==8.2.1
coverage==7.11.0
Cython==3.1.6
cython-lint==0.18.1
distlib==0.4.0
execnet==2.1.1
filelock==3.20.0
gast==0.6.0
hypothesis==6.143.1
idna==3.11
importlib_metadata==8.7.0
iniconfig==2.3.0
json5==0.12.1
meson==1.9.1
meson-python==0.18.0
mpmath==1.3.0
mypy==1.10.0
mypy_extensions==1.1.0
ninja==1.13.0
numpy==2.3.4
packaging==25.0
platformdirs==4.5.0
pluggy==1.6.0
ply==3.11
pooch==1.8.2
pybind11==3.0.1
pycodestyle==2.14.0
Pygments==2.19.2
Pympler==1.1
pyproject-metadata==0.9.1
pyproject_hooks==1.2.0
pytest==8.4.2
pytest-cov==7.0.0
pytest-run-parallel==0.7.1
pytest-timeout==2.4.0
pytest-xdist==3.8.0
pythran==0.18.0
PyYAML==6.0.3
requests==2.32.5
ruff==0.14.3
scipy-openblas32==0.3.30.0.6
setuptools==80.9.0
sortedcontainers==2.4.0
spin==0.15
tabulate==0.9.0
threadpoolctl==3.6.0
tokenize_rt==6.2.0
types-psutil==7.0.0.20251001
typing_extensions==4.15.0
urllib3==2.5.0
virtualenv==20.35.4
zipp==3.23.0

Error message

FAILED: [code=1] scipy/linalg/_linalg_pythran.cpython-313-x86_64-linux-gnu.so.p/_linalg_pythran.cpp 
/opt/hostedtoolcache/Python/3.13.9/x64/bin/pythran -E ../scipy/linalg/_linalg_pythran.py -o scipy/linalg/_linalg_pythran.cpython-313-x86_64-linux-gnu.so.p/_linalg_pythran.cpp
WARNING: Overriding pythran description with argspec information for: numpy.arange
WARNING: Overriding pythran description with argspec information for: numpy.array
WARNING: Overriding pythran description with argspec information for: numpy.concatenate
WARNING: Overriding pythran description with argspec information for: numpy.empty
WARNING: Overriding pythran description with argspec information for: numpy.empty_like
WARNING: Overriding pythran description with argspec information for: numpy.fromiter
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.13.9/x64/bin/pythran", line 7, in <module>
    sys.exit(run())
             ~~~^^
  File "/opt/hostedtoolcache/Python/3.13.9/x64/lib/python3.13/site-packages/pythran/run.py", line 190, in run
    pythran.compile_pythranfile(args.input_file,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.13.9/x64/lib/python3.13/site-packages/pythran/__init__.py", line 127, in __getattr__
    import pythran.toolchain
  File "/opt/hostedtoolcache/Python/3.13.9/x64/lib/python3.13/site-packages/pythran/toolchain.py", line 6, in <module>
    from pythran.backend import Cxx, Python
  File "/opt/hostedtoolcache/Python/3.13.9/x64/lib/python3.13/site-packages/pythran/backend.py", line 7, in <module>
    from pythran.analyses import LocalNodeDeclarations, GlobalDeclarations, Scope
  File "/opt/hostedtoolcache/Python/3.13.9/x64/lib/python3.13/site-packages/pythran/analyses/__init__.py", line 12, in <module>
    from .aliases import Aliases, StrictAliases, InterproceduralAliases
  File "/opt/hostedtoolcache/Python/3.13.9/x64/lib/python3.13/site-packages/pythran/analyses/aliases.py", line 6, in <module>
    from pythran.tables import functions, methods, MODULES
  File "/opt/hostedtoolcache/Python/3.13.9/x64/lib/python3.13/site-packages/pythran/tables.py", line 4676, in <module>
    save_arguments((), MODULES)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.13.9/x64/lib/python3.13/site-packages/pythran/tables.py", line 4611, in save_arguments
    save_arguments(module_name + (elem,), signature)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.13.9/x64/lib/python3.13/site-packages/pythran/tables.py", line 4644, in save_arguments
    defaults += [spec.kwonlydefaults[kw] for kw in spec.kwonlyargs]
                 ~~~~~~~~~~~~~~~~~~~^^^^
KeyError: 'sep'

SciPy/NumPy/Python version and system information

1.17.0.dev0+git20251106.48dba5f 2.3.4 sys.version_info(major=3, minor=14, micro=0, releaselevel='final', serial=0)
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    has ilp64: false
    include directory: /home/nodell/scipy-ft/venv/lib/python3.14t/site-packages/scipy_openblas32/include
    lib directory: /home/nodell/scipy-ft/venv/lib/python3.14t/site-packages/scipy_openblas32/lib
    name: scipy-openblas
    openblas configuration: OpenBLAS 0.3.30 DYNAMIC_ARCH NO_AFFINITY Haswell MAX_THREADS=64
    pc file directory: /home/nodell/scipy-ft
    version: 0.3.30
  lapack:
    detection method: pkgconfig
    found: true
    has ilp64: false
    include directory: /home/nodell/scipy-ft/venv/lib/python3.14t/site-packages/scipy_openblas32/include
    lib directory: /home/nodell/scipy-ft/venv/lib/python3.14t/site-packages/scipy_openblas32/lib
    name: scipy-openblas
    openblas configuration: OpenBLAS 0.3.30 DYNAMIC_ARCH NO_AFFINITY Haswell MAX_THREADS=64
    pc file directory: /home/nodell/scipy-ft
    version: 0.3.30
  pybind11:
    detection method: config-tool
    include directory: unknown
    name: pybind11
    version: 3.0.1
Compilers:
  c:
    commands: cc
    linker: ld.bfd
    name: gcc
    version: 14.2.1
  c++:
    commands: c++
    linker: ld.bfd
    name: gcc
    version: 14.2.1
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 3.1.6
  fortran:
    commands: gfortran
    linker: ld.bfd
    name: gcc
    version: 14.2.1
  pythran:
    include directory: ../venv/lib/python3.14t/site-packages/pythran
    version: 0.18.0
Machine Information:
  build:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
  cross-compiled: false
  host:
    cpu: x86_64
    endian: little
    family: x86_64
    system: linux
Python Information:
  path: /home/nodell/scipy-ft/venv/bin/python
  version: '3.14'

[nickodell]

Looks like it's something that changed inside NumPy. It works at commit e44f8c3395515afdd3ec9f92a1bd90ac216a49b6 and fails at commit 3abf57958fd9d4adc48261359d549ca78de29a44.

[tylerjereddy]

@nickodell I'm bisecting now--if you've already started just tell me to stop

[nickodell]

@nickodell I'm bisecting now--if you've already started just tell me to stop

I have not. One thing I forgot to mention about reproducing this is that you might need to run rm -rf build/scipy/linalg/_linalg_pythran.cpython-314t-x86_64-linux-gnu.so* or use spin build --clean to clear out the generated files. Otherwise it might work even when a clean build will fail.

[WarrenWeckesser]

I just finished bisecting--I didn't see these messages until now. The bad numpy commit is numpy/numpy@32817cc, which is in numpy/numpy#30147.

Ping @jorenham and @serge-sans-paille, in case either has any insight into why that change might cause Pythran to break.

[WarrenWeckesser]

By the way, I can reproduce the Pythran crash outside of the SciPy build with a file like this:

foo.py

#pythran export foo(float64[:], float64[:])
def foo(x, y):
    return x + y

% pythran -E foo.py -o foo.cpp
WARNING: Overriding pythran description with argspec information for: numpy.arange
WARNING: Overriding pythran description with argspec information for: numpy.array
WARNING: Overriding pythran description with argspec information for: numpy.concatenate
WARNING: Overriding pythran description with argspec information for: numpy.empty
WARNING: Overriding pythran description with argspec information for: numpy.empty_like
WARNING: Overriding pythran description with argspec information for: numpy.fromiter
Traceback (most recent call last):
  File "/Users/warren/py3.13.7/bin/pythran", line 7, in <module>
    sys.exit(run())
             ~~~^^
  File "/Users/warren/py3.13.7/lib/python3.13/site-packages/pythran/run.py", line 190, in run
    pythran.compile_pythranfile(args.input_file,
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/warren/py3.13.7/lib/python3.13/site-packages/pythran/__init__.py", line 127, in __getattr__
    import pythran.toolchain
  File "/Users/warren/py3.13.7/lib/python3.13/site-packages/pythran/toolchain.py", line 6, in <module>
    from pythran.backend import Cxx, Python
  File "/Users/warren/py3.13.7/lib/python3.13/site-packages/pythran/backend.py", line 7, in <module>
    from pythran.analyses import LocalNodeDeclarations, GlobalDeclarations, Scope
  File "/Users/warren/py3.13.7/lib/python3.13/site-packages/pythran/analyses/__init__.py", line 12, in <module>
    from .aliases import Aliases, StrictAliases, InterproceduralAliases
  File "/Users/warren/py3.13.7/lib/python3.13/site-packages/pythran/analyses/aliases.py", line 6, in <module>
    from pythran.tables import functions, methods, MODULES
  File "/Users/warren/py3.13.7/lib/python3.13/site-packages/pythran/tables.py", line 4676, in <module>
    save_arguments((), MODULES)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/Users/warren/py3.13.7/lib/python3.13/site-packages/pythran/tables.py", line 4611, in save_arguments
    save_arguments(module_name + (elem,), signature)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/warren/py3.13.7/lib/python3.13/site-packages/pythran/tables.py", line 4644, in save_arguments
    defaults += [spec.kwonlydefaults[kw] for kw in spec.kwonlyargs]
                 ~~~~~~~~~~~~~~~~~~~^^^^
KeyError: 'sep'

[WarrenWeckesser]

See serge-sans-paille/pythran#2368; @stefanv already reported this upstream because scikit-image ran into the same issue, and @serge-sans-paille is already working on a fix.

[nickodell]

@WarrenWeckesser Good detective work!

Thanks for your help in figuring that out, even if it was already reported. (Damn, scooped by two hours!)

[jorenham]

Last weeks I have been fixing a lot of numpy functions, classes, and methods, so that they will no longer cause a ValueError to be raised when passed to inspect.signature, and numpy/numpy#30147 is one of those PR's.

Judging by serge-sans-paille/pythran#2368 (comment) this seems to indeed have been caused by the numpy.tostring signature. It includes a keyword-only parameter, sep, without a default. It has been like that since NumPy 1.14, but only now will that also be visible through inspect.signature. So I suppose this is a bug / missing feature in Pythran, rather than a NumPy regression. And unfortunately I can't see any way that we can do on the NumPy side to get around this issue (or at least not without reverting).

[serge-sans-paille]

I'm working on it in serge-sans-paille/pythran#2369
The only thing that may be a numpy's issue is the fact that sep is advertised as not having a default, while it actually has one (which raises an exception, so it's actually quite unclear)

>>> import numpy as np, inspect
>>> inspect.getfullargspec(np.fromstring)
FullArgSpec(args=['string', 'dtype', 'count'], varargs=None, varkw=None, defaults=(None, -1), kwonlyargs=['sep', 'like'], kwonlydefaults={'like': None}, annotations={})
>>> np.fromstring("1")
Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    np.fromstring("1")
    ~~~~~~~~~~~~~^^^^^
ValueError: The binary mode of fromstring is removed, use frombuffer instead

[jorenham]

The only thing that may be a numpy's issue is the fact that sep is advertised as not having a default, while it actually has one (which raises an exception, so it's actually quite unclear)

Yea it has been like that since NumPy 1.14. This is also explained in its docstring:

sep : str, optional
    The string separating numbers in the data; extra whitespace between
    elements is also ignored.

    .. deprecated:: 1.14
        Passing ``sep=''``, the default, is deprecated since it will
        trigger the deprecated binary mode of this function. This mode
        interprets `string` as binary bytes, rather than ASCII text with
        decimal numbers, an operation which is better spelt
        ``frombuffer(string, dtype, count)``. If `string` contains unicode
        text, the binary mode of `fromstring` will first encode it into
        bytes using utf-8, which will not produce sane results.

Which effectively makes sep a required keyword-only argument. So in my view, it's the docstring that's inaccurate here, not the inspect signature. An important purpose of that signature is to prevent users from triggering errors at runtime, and that's what this achieves.
But yea, it's annoyingly confusing...

See also numpy/numpy#27038