[BUG] float(string): error: invalid cast from type ‘std::string’ to type ‘PyObject*’

[hroncok]

Describe the bug

I am trying to move from Cython 0.29 to Cython 3 in MUSIC, INCF/MUSIC#77

I hit an interesting problem that I was able to reproduce isolated.

A C++ string cannot be converted via float(string) call in Cython 3. This wokred in Cython 0.29.

Code to reproduce the behaviour:

reproducer.pxd

from libcpp.string cimport string

reproducer.pyx

# distutils: language = c++

def reproduce():
    cdef string vs
    return float(vs)

setup.py

from setuptools import setup
from Cython.Build import cythonize
setup(ext_modules=cythonize("reproducer.pyx"))

$ python setup.py build_ext --inplace

Expected behaviour

With Cython 0.29.36, this compiles and behaves reasonably. When no value is set (as in the reproducer), the code executes as:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "reproducer.pyx", line 5, in reproducer.reproduce
    return float(vs)
ValueError: could not convert string to float: b''

When I change the reproducer to have cdef string vs = "3.14", the function returns a 3.14 Python float.

---

With Cython 3.0.5, it does not compile:

$ python setup.py build_ext --inplace
Compiling reproducer.pyx because it changed.
[1/1] Cythonizing reproducer.pyx
.../cython-string-float/__venv3__/lib64/python3.11/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: .../cython-string-float/reproducer.pxd
  tree = Parsing.p_module(s, pxd, full_module_name)
running build_ext
building 'reproducer' extension
gcc -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I.../cython-string-float/__venv3__/include -I/usr/include/python3.11 -c reproducer.cpp -o build/temp.linux-x86_64-cpython-311/reproducer.o
reproducer.cpp: In function ‘PyObject* __pyx_pf_10reproducer_reproduce(PyObject*)’:
reproducer.cpp:2669:40: error: invalid cast from type ‘std::string’ {aka ‘std::__cxx11::basic_string<char>’} to type ‘PyObject*’ {aka ‘_object*’}
 2669 |   __pyx_t_1 = __Pyx_PyObject_AsDouble(((PyObject *)__pyx_v_vs)); if (unlikely(__pyx_t_1 == ((double)((double)-1)) && PyErr_Occurred())) __PYX_ERR(1, 5, __pyx_L1_error)
      |                                        ^~~~~~~~~~~~~~~~~~~~~~
reproducer.cpp:1465:43: note: in definition of macro ‘likely’
 1465 |   #define likely(x)   __builtin_expect(!!(x), 1)
      |                                           ^
/usr/include/python3.11/object.h:107:28: note: in expansion of macro ‘_Py_CAST’
  107 | #define _PyObject_CAST(op) _Py_CAST(PyObject*, (op))
      |                            ^~~~~~~~
/usr/include/python3.11/object.h:153:43: note: in expansion of macro ‘_PyObject_CAST’
  153 | #  define Py_IS_TYPE(ob, type) Py_IS_TYPE(_PyObject_CAST(ob), type)
      |                                           ^~~~~~~~~~~~~~
/usr/include/python3.11/floatobject.h:17:32: note: in expansion of macro ‘Py_IS_TYPE’
   17 | #define PyFloat_CheckExact(op) Py_IS_TYPE(op, &PyFloat_Type)
      |                                ^~~~~~~~~~
reproducer.cpp:1843:10: note: in expansion of macro ‘PyFloat_CheckExact’
 1843 | ((likely(PyFloat_CheckExact(obj))) ?  PyFloat_AS_DOUBLE(obj) :\
      |          ^~~~~~~~~~~~~~~~~~
reproducer.cpp:2669:15: note: in expansion of macro ‘__Pyx_PyObject_AsDouble’
 2669 |   __pyx_t_1 = __Pyx_PyObject_AsDouble(((PyObject *)__pyx_v_vs)); if (unlikely(__pyx_t_1 == ((double)((double)-1)) && PyErr_Occurred())) __PYX_ERR(1, 5, __pyx_L1_error)
      |               ^~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/python3.11/floatobject.h:47,
                 from /usr/include/python3.11/Python.h:55,
                 from reproducer.cpp:30:
reproducer.cpp:2669:40: error: invalid cast from type ‘std::string’ {aka ‘std::__cxx11::basic_string<char>’} to type ‘PyObject*’ {aka ‘_object*’}
 2669 |   __pyx_t_1 = __Pyx_PyObject_AsDouble(((PyObject *)__pyx_v_vs)); if (unlikely(__pyx_t_1 == ((double)((double)-1)) && PyErr_Occurred())) __PYX_ERR(1, 5, __pyx_L1_error)
      |                                        ^~~~~~~~~~~~~~~~~~~~~~
/usr/include/python3.11/cpython/floatobject.h:12:51: note: in definition of macro ‘PyFloat_AS_DOUBLE’
   12 | #define PyFloat_AS_DOUBLE(op) (((PyFloatObject *)(op))->ob_fval)
      |                                                   ^~
reproducer.cpp:2669:15: note: in expansion of macro ‘__Pyx_PyObject_AsDouble’
 2669 |   __pyx_t_1 = __Pyx_PyObject_AsDouble(((PyObject *)__pyx_v_vs)); if (unlikely(__pyx_t_1 == ((double)((double)-1)) && PyErr_Occurred())) __PYX_ERR(1, 5, __pyx_L1_error)
      |               ^~~~~~~~~~~~~~~~~~~~~~~
reproducer.cpp:2669:40: error: invalid cast from type ‘std::string’ {aka ‘std::__cxx11::basic_string<char>’} to type ‘PyObject*’ {aka ‘_object*’}
 2669 |   __pyx_t_1 = __Pyx_PyObject_AsDouble(((PyObject *)__pyx_v_vs)); if (unlikely(__pyx_t_1 == ((double)((double)-1)) && PyErr_Occurred())) __PYX_ERR(1, 5, __pyx_L1_error)
      |                                        ^~~~~~~~~~~~~~~~~~~~~~
reproducer.cpp:1465:43: note: in definition of macro ‘likely’
 1465 |   #define likely(x)   __builtin_expect(!!(x), 1)
      |                                           ^
/usr/include/python3.11/object.h:107:28: note: in expansion of macro ‘_Py_CAST’
  107 | #define _PyObject_CAST(op) _Py_CAST(PyObject*, (op))
      |                            ^~~~~~~~
/usr/include/python3.11/object.h:153:43: note: in expansion of macro ‘_PyObject_CAST’
  153 | #  define Py_IS_TYPE(ob, type) Py_IS_TYPE(_PyObject_CAST(ob), type)
      |                                           ^~~~~~~~~~~~~~
/usr/include/python3.11/longobject.h:14:31: note: in expansion of macro ‘Py_IS_TYPE’
   14 | #define PyLong_CheckExact(op) Py_IS_TYPE(op, &PyLong_Type)
      |                               ^~~~~~~~~~
reproducer.cpp:1844:9: note: in expansion of macro ‘PyLong_CheckExact’
 1844 |  likely(PyLong_CheckExact(obj)) ?\
      |         ^~~~~~~~~~~~~~~~~
reproducer.cpp:2669:15: note: in expansion of macro ‘__Pyx_PyObject_AsDouble’
 2669 |   __pyx_t_1 = __Pyx_PyObject_AsDouble(((PyObject *)__pyx_v_vs)); if (unlikely(__pyx_t_1 == ((double)((double)-1)) && PyErr_Occurred())) __PYX_ERR(1, 5, __pyx_L1_error)
      |               ^~~~~~~~~~~~~~~~~~~~~~~
reproducer.cpp:2669:40: error: invalid cast from type ‘std::string’ {aka ‘std::__cxx11::basic_string<char>’} to type ‘PyObject*’ {aka ‘_object*’}
 2669 |   __pyx_t_1 = __Pyx_PyObject_AsDouble(((PyObject *)__pyx_v_vs)); if (unlikely(__pyx_t_1 == ((double)((double)-1)) && PyErr_Occurred())) __PYX_ERR(1, 5, __pyx_L1_error)
      |                                        ^~~~~~~~~~~~~~~~~~~~~~
reproducer.cpp:1845:18: note: in definition of macro ‘__Pyx_PyObject_AsDouble’
 1845 |  PyLong_AsDouble(obj) : __Pyx__PyObject_AsDouble(obj))
      |                  ^~~
reproducer.cpp:2669:40: error: invalid cast from type ‘std::string’ {aka ‘std::__cxx11::basic_string<char>’} to type ‘PyObject*’ {aka ‘_object*’}
 2669 |   __pyx_t_1 = __Pyx_PyObject_AsDouble(((PyObject *)__pyx_v_vs)); if (unlikely(__pyx_t_1 == ((double)((double)-1)) && PyErr_Occurred())) __PYX_ERR(1, 5, __pyx_L1_error)
      |                                        ^~~~~~~~~~~~~~~~~~~~~~
reproducer.cpp:1845:50: note: in definition of macro ‘__Pyx_PyObject_AsDouble’
 1845 |  PyLong_AsDouble(obj) : __Pyx__PyObject_AsDouble(obj))
      |                                                  ^~~
reproducer.cpp: At global scope:
reproducer.cpp:3799:15: warning: ‘double __Pyx__PyObject_AsDouble(PyObject*)’ defined but not used [-Wunused-function]
 3799 | static double __Pyx__PyObject_AsDouble(PyObject* obj) {
      |               ^~~~~~~~~~~~~~~~~~~~~~~~
error: command '/usr/lib64/ccache/gcc' failed with exit code 1

OS

Fedora Linux 37 to 40

Python version

3.11.6 and 3.12.0

Cython version

3.0.5

Additional context

No response

[da-woods]

Bisects to c59a1dc

[scoder]

Fixed. However, the best thing to do for 3.1 is to optimise float(std::string). I'll look into that as well.

[hroncok]

Thanks