New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Py_buffer builtin struct unusable after 0.27.0 #2046

Open
klaussfreire opened this Issue Dec 18, 2017 · 5 comments

Comments

Projects
None yet
2 participants
@klaussfreire
Contributor

klaussfreire commented Dec 18, 2017

Since 0.27.1, declaring Py_buffer variables in pure-python mode ceased working. This applies to 0.27.1, 0.27.2 and 0.27.3.

import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    return cython.cast(int, cython.address(pybuf))

print f()

I know this isn't really useful code, but it illustrates the issue with a minimal example. I'm getting this error in actual working (with 0.27.0) code.

When compiling with cython 0.27.2 (and .3), gives:

(cython)claudiofreire@klaumpptophp:~> cython buffer_bug.py

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
                      ^
------------------------------------------------------------

buffer_bug.py:3:23: Not a type

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    return cython.cast(int, cython.address(pybuf))
                                          ^
------------------------------------------------------------

buffer_bug.py:5:43: undeclared name not builtin: pybuf

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    return cython.cast(int, cython.address(pybuf))
                                 ^
------------------------------------------------------------

buffer_bug.py:5:34: Taking address of non-lvalue (type Python object)

In cython 0.27.0, however, it builds correctly, generating the following code:

  __pyx_t_1 = __Pyx_PyInt_From_int(((int)(&__pyx_v_pybuf))); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 5, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  __pyx_r = __pyx_t_1;
  __pyx_t_1 = 0;
  goto __pyx_L0;

I could find no directly relatable changes in the commit history, but I'm not that familiar with this code, so maybe someone else can figure this out.

To clarify, in non-pure-python mode it works correctly:

from cpython cimport Py_buffer

def f():
    cdef Py_buffer pybuf
    return <int>&pybuf

print f()

Compiles in all versions.

Adding the cimport to buffer_bug.pxd in pure-python mode doesn't change a thing.

FTR, I realize Py_buffer is unusable in pure python. The use case for this always involves compiled-only code paths:

if cython.compiled:
    # do something with Py_buffer
else:
    # do it the slower pure-python way
@scoder

This comment has been minimized.

Show comment
Hide comment
@scoder

scoder Jan 12, 2018

Contributor

The pybuf variable is not actually defined in your example. According to Python semantics, a local variable does not exist before it is assigned. Is that related, or do you also see this in code that correctly creates your variable?

Contributor

scoder commented Jan 12, 2018

The pybuf variable is not actually defined in your example. According to Python semantics, a local variable does not exist before it is assigned. Is that related, or do you also see this in code that correctly creates your variable?

@klaussfreire

This comment has been minimized.

Show comment
Hide comment
@klaussfreire

klaussfreire Jan 12, 2018

Contributor

You cannot assign a Py_buffer, as it is a struct intended to be used by passing pointers to it.

Thus, most real-world use cases will not contain assignments.

That said, adding no-op assignments to make it local without actually executing the assignment doesn't work either:

import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    pybuf = pybuf
    return cython.cast(int, cython.address(pybuf))

print f()
Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
                      ^
------------------------------------------------------------

buffer_bug.py:3:23: Not a type

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    pybuf = pybuf
           ^
------------------------------------------------------------

buffer_bug.py:5:12: local variable 'pybuf' referenced before assignment

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    pybuf = pybuf
    return cython.cast(int, cython.address(pybuf))
                                 ^
------------------------------------------------------------

buffer_bug.py:6:34: Cannot take address of Python variable 'pybuf'

Using an alternative declaration syntax doesn't work either in 0.27.3, yet it does in 0.27.0:

import cython

def f():
    pybuf = cython.declare('Py_buffer')
    return cython.cast(int, cython.address(pybuf))

print f()
Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
                          ^
------------------------------------------------------------

buffer_bug.py:4:27: Unknown type

Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
                 ^
------------------------------------------------------------

buffer_bug.py:4:18: 'declare' not a valid cython language construct

Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
                 ^
------------------------------------------------------------

buffer_bug.py:4:18: 'declare' not a valid cython attribute or is being used incorrectly

Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
   ^
------------------------------------------------------------

buffer_bug.py:4:4: Compiler crash in AnalyseExpressionsTransform

ModuleNode.body = StatListNode(buffer_bug.py:1:0)
StatListNode.stats[1] = StatListNode(buffer_bug.py:3:0)
StatListNode.stats[0] = DefNode(buffer_bug.py:3:0,
    is_cyfunction = True,
    modifiers = [...]/0,
    name = u'f',
    np_args_idx = [...]/0,
    py_wrapper_required = True,
    reqd_kw_flags_cname = '0',
    used = True)
File 'Nodes.py', line 422, in analyse_expressions: StatListNode(buffer_bug.py:4:4,
    is_terminator = True)
File 'Nodes.py', line 4863, in analyse_expressions: SingleAssignmentNode(buffer_bug.py:4:26)
File 'Nodes.py', line 4989, in analyse_types: SingleAssignmentNode(buffer_bug.py:4:26)
File 'ExprNodes.py', line 1944, in analyse_target_types: NameNode(buffer_bug.py:4:4,
    cf_maybe_null = True,
    is_name = True,
    name = u'pybuf',
    result_is_used = True,
    use_managed_ref = True)
File 'ExprNodes.py', line 2003, in analyse_entry: NameNode(buffer_bug.py:4:4,
    cf_maybe_null = True,
    is_name = True,
    name = u'pybuf',
    result_is_used = True,
    use_managed_ref = True)
File 'ExprNodes.py', line 2017, in check_identifier_kind: NameNode(buffer_bug.py:4:4,
    cf_maybe_null = True,
    is_name = True,
    name = u'pybuf',
    result_is_used = True,
    use_managed_ref = True)

Compiler crash traceback from this point on:
  File "/home/claudiofreire/venv/cython/lib/python2.7/site-packages/Cython/Compiler/ExprNodes.py", line 2017, in check_identifier_kind
    if entry.is_type and entry.type.is_extension_type:
AttributeError: 'NoneType' object has no attribute 'is_type'
Contributor

klaussfreire commented Jan 12, 2018

You cannot assign a Py_buffer, as it is a struct intended to be used by passing pointers to it.

Thus, most real-world use cases will not contain assignments.

That said, adding no-op assignments to make it local without actually executing the assignment doesn't work either:

import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    pybuf = pybuf
    return cython.cast(int, cython.address(pybuf))

print f()
Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
                      ^
------------------------------------------------------------

buffer_bug.py:3:23: Not a type

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    pybuf = pybuf
           ^
------------------------------------------------------------

buffer_bug.py:5:12: local variable 'pybuf' referenced before assignment

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    pybuf = pybuf
    return cython.cast(int, cython.address(pybuf))
                                 ^
------------------------------------------------------------

buffer_bug.py:6:34: Cannot take address of Python variable 'pybuf'

Using an alternative declaration syntax doesn't work either in 0.27.3, yet it does in 0.27.0:

import cython

def f():
    pybuf = cython.declare('Py_buffer')
    return cython.cast(int, cython.address(pybuf))

print f()
Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
                          ^
------------------------------------------------------------

buffer_bug.py:4:27: Unknown type

Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
                 ^
------------------------------------------------------------

buffer_bug.py:4:18: 'declare' not a valid cython language construct

Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
                 ^
------------------------------------------------------------

buffer_bug.py:4:18: 'declare' not a valid cython attribute or is being used incorrectly

Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
   ^
------------------------------------------------------------

buffer_bug.py:4:4: Compiler crash in AnalyseExpressionsTransform

ModuleNode.body = StatListNode(buffer_bug.py:1:0)
StatListNode.stats[1] = StatListNode(buffer_bug.py:3:0)
StatListNode.stats[0] = DefNode(buffer_bug.py:3:0,
    is_cyfunction = True,
    modifiers = [...]/0,
    name = u'f',
    np_args_idx = [...]/0,
    py_wrapper_required = True,
    reqd_kw_flags_cname = '0',
    used = True)
File 'Nodes.py', line 422, in analyse_expressions: StatListNode(buffer_bug.py:4:4,
    is_terminator = True)
File 'Nodes.py', line 4863, in analyse_expressions: SingleAssignmentNode(buffer_bug.py:4:26)
File 'Nodes.py', line 4989, in analyse_types: SingleAssignmentNode(buffer_bug.py:4:26)
File 'ExprNodes.py', line 1944, in analyse_target_types: NameNode(buffer_bug.py:4:4,
    cf_maybe_null = True,
    is_name = True,
    name = u'pybuf',
    result_is_used = True,
    use_managed_ref = True)
File 'ExprNodes.py', line 2003, in analyse_entry: NameNode(buffer_bug.py:4:4,
    cf_maybe_null = True,
    is_name = True,
    name = u'pybuf',
    result_is_used = True,
    use_managed_ref = True)
File 'ExprNodes.py', line 2017, in check_identifier_kind: NameNode(buffer_bug.py:4:4,
    cf_maybe_null = True,
    is_name = True,
    name = u'pybuf',
    result_is_used = True,
    use_managed_ref = True)

Compiler crash traceback from this point on:
  File "/home/claudiofreire/venv/cython/lib/python2.7/site-packages/Cython/Compiler/ExprNodes.py", line 2017, in check_identifier_kind
    if entry.is_type and entry.type.is_extension_type:
AttributeError: 'NoneType' object has no attribute 'is_type'
@klaussfreire

This comment has been minimized.

Show comment
Hide comment
@klaussfreire

klaussfreire Jan 12, 2018

Contributor

Real-world code can be found here

Contributor

klaussfreire commented Jan 12, 2018

Real-world code can be found here

@klaussfreire

This comment has been minimized.

Show comment
Hide comment
@klaussfreire

klaussfreire Jun 8, 2018

Contributor

This still happens in 0.28.3

Contributor

klaussfreire commented Jun 8, 2018

This still happens in 0.28.3

@klaussfreire

This comment has been minimized.

Show comment
Hide comment
@klaussfreire

klaussfreire Jun 8, 2018

Contributor

Create #2317 which seems to fix this

Contributor

klaussfreire commented Jun 8, 2018

Create #2317 which seems to fix this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment