Py_buffer builtin struct unusable after 0.27.0

[klaussfreire]

Since 0.27.1, declaring Py_buffer variables in pure-python mode ceased working. This applies to 0.27.1, 0.27.2 and 0.27.3.

import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    return cython.cast(int, cython.address(pybuf))

print f()

I know this isn't really useful code, but it illustrates the issue with a minimal example. I'm getting this error in actual working (with 0.27.0) code.

When compiling with cython 0.27.2 (and .3), gives:

(cython)claudiofreire@klaumpptophp:~> cython buffer_bug.py

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
                      ^
------------------------------------------------------------

buffer_bug.py:3:23: Not a type

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    return cython.cast(int, cython.address(pybuf))
                                          ^
------------------------------------------------------------

buffer_bug.py:5:43: undeclared name not builtin: pybuf

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    return cython.cast(int, cython.address(pybuf))
                                 ^
------------------------------------------------------------

buffer_bug.py:5:34: Taking address of non-lvalue (type Python object)

In cython 0.27.0, however, it builds correctly, generating the following code:

  __pyx_t_1 = __Pyx_PyInt_From_int(((int)(&__pyx_v_pybuf))); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 5, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  __pyx_r = __pyx_t_1;
  __pyx_t_1 = 0;
  goto __pyx_L0;

I could find no directly relatable changes in the commit history, but I'm not that familiar with this code, so maybe someone else can figure this out.

To clarify, in non-pure-python mode it works correctly:

from cpython cimport Py_buffer

def f():
    cdef Py_buffer pybuf
    return <int>&pybuf

print f()

Compiles in all versions.

Adding the cimport to buffer_bug.pxd in pure-python mode doesn't change a thing.

FTR, I realize Py_buffer is unusable in pure python. The use case for this always involves compiled-only code paths:

if cython.compiled:
    # do something with Py_buffer
else:
    # do it the slower pure-python way

[scoder]

The pybuf variable is not actually defined in your example. According to Python semantics, a local variable does not exist before it is assigned. Is that related, or do you also see this in code that correctly creates your variable?

[scoder]

The pybuf variable is not actually defined in your example. According to Python semantics, a local variable does not exist before it is assigned. Is that related, or do you also see this in code that correctly creates your variable?

[klaussfreire]

You cannot assign a Py_buffer, as it is a struct intended to be used by passing pointers to it.

Thus, most real-world use cases will not contain assignments.

That said, adding no-op assignments to make it local without actually executing the assignment doesn't work either:

import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    pybuf = pybuf
    return cython.cast(int, cython.address(pybuf))

print f()

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
                      ^
------------------------------------------------------------

buffer_bug.py:3:23: Not a type

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    pybuf = pybuf
           ^
------------------------------------------------------------

buffer_bug.py:5:12: local variable 'pybuf' referenced before assignment

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    pybuf = pybuf
    return cython.cast(int, cython.address(pybuf))
                                 ^
------------------------------------------------------------

buffer_bug.py:6:34: Cannot take address of Python variable 'pybuf'

Using an alternative declaration syntax doesn't work either in 0.27.3, yet it does in 0.27.0:

import cython

def f():
    pybuf = cython.declare('Py_buffer')
    return cython.cast(int, cython.address(pybuf))

print f()

Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
                          ^
------------------------------------------------------------

buffer_bug.py:4:27: Unknown type

Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
                 ^
------------------------------------------------------------

buffer_bug.py:4:18: 'declare' not a valid cython language construct

Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
                 ^
------------------------------------------------------------

buffer_bug.py:4:18: 'declare' not a valid cython attribute or is being used incorrectly

Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
   ^
------------------------------------------------------------

buffer_bug.py:4:4: Compiler crash in AnalyseExpressionsTransform

ModuleNode.body = StatListNode(buffer_bug.py:1:0)
StatListNode.stats[1] = StatListNode(buffer_bug.py:3:0)
StatListNode.stats[0] = DefNode(buffer_bug.py:3:0,
    is_cyfunction = True,
    modifiers = [...]/0,
    name = u'f',
    np_args_idx = [...]/0,
    py_wrapper_required = True,
    reqd_kw_flags_cname = '0',
    used = True)
File 'Nodes.py', line 422, in analyse_expressions: StatListNode(buffer_bug.py:4:4,
    is_terminator = True)
File 'Nodes.py', line 4863, in analyse_expressions: SingleAssignmentNode(buffer_bug.py:4:26)
File 'Nodes.py', line 4989, in analyse_types: SingleAssignmentNode(buffer_bug.py:4:26)
File 'ExprNodes.py', line 1944, in analyse_target_types: NameNode(buffer_bug.py:4:4,
    cf_maybe_null = True,
    is_name = True,
    name = u'pybuf',
    result_is_used = True,
    use_managed_ref = True)
File 'ExprNodes.py', line 2003, in analyse_entry: NameNode(buffer_bug.py:4:4,
    cf_maybe_null = True,
    is_name = True,
    name = u'pybuf',
    result_is_used = True,
    use_managed_ref = True)
File 'ExprNodes.py', line 2017, in check_identifier_kind: NameNode(buffer_bug.py:4:4,
    cf_maybe_null = True,
    is_name = True,
    name = u'pybuf',
    result_is_used = True,
    use_managed_ref = True)

Compiler crash traceback from this point on:
  File "/home/claudiofreire/venv/cython/lib/python2.7/site-packages/Cython/Compiler/ExprNodes.py", line 2017, in check_identifier_kind
    if entry.is_type and entry.type.is_extension_type:
AttributeError: 'NoneType' object has no attribute 'is_type'

[klaussfreire]

You cannot assign a Py_buffer, as it is a struct intended to be used by passing pointers to it.

Thus, most real-world use cases will not contain assignments.

That said, adding no-op assignments to make it local without actually executing the assignment doesn't work either:

import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    pybuf = pybuf
    return cython.cast(int, cython.address(pybuf))

print f()

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
                      ^
------------------------------------------------------------

buffer_bug.py:3:23: Not a type

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    pybuf = pybuf
           ^
------------------------------------------------------------

buffer_bug.py:5:12: local variable 'pybuf' referenced before assignment

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    pybuf = pybuf
    return cython.cast(int, cython.address(pybuf))
                                 ^
------------------------------------------------------------

buffer_bug.py:6:34: Cannot take address of Python variable 'pybuf'

Using an alternative declaration syntax doesn't work either in 0.27.3, yet it does in 0.27.0:

import cython

def f():
    pybuf = cython.declare('Py_buffer')
    return cython.cast(int, cython.address(pybuf))

print f()

Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
                          ^
------------------------------------------------------------

buffer_bug.py:4:27: Unknown type

Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
                 ^
------------------------------------------------------------

buffer_bug.py:4:18: 'declare' not a valid cython language construct

Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
                 ^
------------------------------------------------------------

buffer_bug.py:4:18: 'declare' not a valid cython attribute or is being used incorrectly

Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
   ^
------------------------------------------------------------

buffer_bug.py:4:4: Compiler crash in AnalyseExpressionsTransform

ModuleNode.body = StatListNode(buffer_bug.py:1:0)
StatListNode.stats[1] = StatListNode(buffer_bug.py:3:0)
StatListNode.stats[0] = DefNode(buffer_bug.py:3:0,
    is_cyfunction = True,
    modifiers = [...]/0,
    name = u'f',
    np_args_idx = [...]/0,
    py_wrapper_required = True,
    reqd_kw_flags_cname = '0',
    used = True)
File 'Nodes.py', line 422, in analyse_expressions: StatListNode(buffer_bug.py:4:4,
    is_terminator = True)
File 'Nodes.py', line 4863, in analyse_expressions: SingleAssignmentNode(buffer_bug.py:4:26)
File 'Nodes.py', line 4989, in analyse_types: SingleAssignmentNode(buffer_bug.py:4:26)
File 'ExprNodes.py', line 1944, in analyse_target_types: NameNode(buffer_bug.py:4:4,
    cf_maybe_null = True,
    is_name = True,
    name = u'pybuf',
    result_is_used = True,
    use_managed_ref = True)
File 'ExprNodes.py', line 2003, in analyse_entry: NameNode(buffer_bug.py:4:4,
    cf_maybe_null = True,
    is_name = True,
    name = u'pybuf',
    result_is_used = True,
    use_managed_ref = True)
File 'ExprNodes.py', line 2017, in check_identifier_kind: NameNode(buffer_bug.py:4:4,
    cf_maybe_null = True,
    is_name = True,
    name = u'pybuf',
    result_is_used = True,
    use_managed_ref = True)

Compiler crash traceback from this point on:
  File "/home/claudiofreire/venv/cython/lib/python2.7/site-packages/Cython/Compiler/ExprNodes.py", line 2017, in check_identifier_kind
    if entry.is_type and entry.type.is_extension_type:
AttributeError: 'NoneType' object has no attribute 'is_type'

[klaussfreire]

Real-world code can be found here

[klaussfreire]

Real-world code can be found here

[klaussfreire]

This still happens in 0.28.3

[klaussfreire]

This still happens in 0.28.3

[klaussfreire]

Create #2317 which seems to fix this

[klaussfreire]

Create #2317 which seems to fix this