Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Py_buffer builtin struct unusable after 0.27.0 #2046

Closed
klaussfreire opened this issue Dec 18, 2017 · 7 comments
Closed

Py_buffer builtin struct unusable after 0.27.0 #2046

klaussfreire opened this issue Dec 18, 2017 · 7 comments

Comments

@klaussfreire
Copy link
Contributor

klaussfreire commented Dec 18, 2017

Since 0.27.1, declaring Py_buffer variables in pure-python mode ceased working. This applies to 0.27.1, 0.27.2 and 0.27.3.

import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    return cython.cast(int, cython.address(pybuf))

print f()

I know this isn't really useful code, but it illustrates the issue with a minimal example. I'm getting this error in actual working (with 0.27.0) code.

When compiling with cython 0.27.2 (and .3), gives:

(cython)claudiofreire@klaumpptophp:~> cython buffer_bug.py

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
                      ^
------------------------------------------------------------

buffer_bug.py:3:23: Not a type

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    return cython.cast(int, cython.address(pybuf))
                                          ^
------------------------------------------------------------

buffer_bug.py:5:43: undeclared name not builtin: pybuf

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    return cython.cast(int, cython.address(pybuf))
                                 ^
------------------------------------------------------------

buffer_bug.py:5:34: Taking address of non-lvalue (type Python object)

In cython 0.27.0, however, it builds correctly, generating the following code:

  __pyx_t_1 = __Pyx_PyInt_From_int(((int)(&__pyx_v_pybuf))); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 5, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  __pyx_r = __pyx_t_1;
  __pyx_t_1 = 0;
  goto __pyx_L0;

I could find no directly relatable changes in the commit history, but I'm not that familiar with this code, so maybe someone else can figure this out.

To clarify, in non-pure-python mode it works correctly:

from cpython cimport Py_buffer

def f():
    cdef Py_buffer pybuf
    return <int>&pybuf

print f()

Compiles in all versions.

Adding the cimport to buffer_bug.pxd in pure-python mode doesn't change a thing.

FTR, I realize Py_buffer is unusable in pure python. The use case for this always involves compiled-only code paths:

if cython.compiled:
    # do something with Py_buffer
else:
    # do it the slower pure-python way
@scoder
Copy link
Contributor

scoder commented Jan 12, 2018

The pybuf variable is not actually defined in your example. According to Python semantics, a local variable does not exist before it is assigned. Is that related, or do you also see this in code that correctly creates your variable?

@klaussfreire
Copy link
Contributor Author

klaussfreire commented Jan 12, 2018

You cannot assign a Py_buffer, as it is a struct intended to be used by passing pointers to it.

Thus, most real-world use cases will not contain assignments.

That said, adding no-op assignments to make it local without actually executing the assignment doesn't work either:

import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    pybuf = pybuf
    return cython.cast(int, cython.address(pybuf))

print f()
Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
                      ^
------------------------------------------------------------

buffer_bug.py:3:23: Not a type

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    pybuf = pybuf
           ^
------------------------------------------------------------

buffer_bug.py:5:12: local variable 'pybuf' referenced before assignment

Error compiling Cython file:
------------------------------------------------------------
...
import cython

@cython.locals(pybuf = 'Py_buffer')
def f():
    pybuf = pybuf
    return cython.cast(int, cython.address(pybuf))
                                 ^
------------------------------------------------------------

buffer_bug.py:6:34: Cannot take address of Python variable 'pybuf'

Using an alternative declaration syntax doesn't work either in 0.27.3, yet it does in 0.27.0:

import cython

def f():
    pybuf = cython.declare('Py_buffer')
    return cython.cast(int, cython.address(pybuf))

print f()
Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
                          ^
------------------------------------------------------------

buffer_bug.py:4:27: Unknown type

Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
                 ^
------------------------------------------------------------

buffer_bug.py:4:18: 'declare' not a valid cython language construct

Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
                 ^
------------------------------------------------------------

buffer_bug.py:4:18: 'declare' not a valid cython attribute or is being used incorrectly

Error compiling Cython file:
------------------------------------------------------------
...
import cython

def f():
    pybuf = cython.declare('Py_buffer')
   ^
------------------------------------------------------------

buffer_bug.py:4:4: Compiler crash in AnalyseExpressionsTransform

ModuleNode.body = StatListNode(buffer_bug.py:1:0)
StatListNode.stats[1] = StatListNode(buffer_bug.py:3:0)
StatListNode.stats[0] = DefNode(buffer_bug.py:3:0,
    is_cyfunction = True,
    modifiers = [...]/0,
    name = u'f',
    np_args_idx = [...]/0,
    py_wrapper_required = True,
    reqd_kw_flags_cname = '0',
    used = True)
File 'Nodes.py', line 422, in analyse_expressions: StatListNode(buffer_bug.py:4:4,
    is_terminator = True)
File 'Nodes.py', line 4863, in analyse_expressions: SingleAssignmentNode(buffer_bug.py:4:26)
File 'Nodes.py', line 4989, in analyse_types: SingleAssignmentNode(buffer_bug.py:4:26)
File 'ExprNodes.py', line 1944, in analyse_target_types: NameNode(buffer_bug.py:4:4,
    cf_maybe_null = True,
    is_name = True,
    name = u'pybuf',
    result_is_used = True,
    use_managed_ref = True)
File 'ExprNodes.py', line 2003, in analyse_entry: NameNode(buffer_bug.py:4:4,
    cf_maybe_null = True,
    is_name = True,
    name = u'pybuf',
    result_is_used = True,
    use_managed_ref = True)
File 'ExprNodes.py', line 2017, in check_identifier_kind: NameNode(buffer_bug.py:4:4,
    cf_maybe_null = True,
    is_name = True,
    name = u'pybuf',
    result_is_used = True,
    use_managed_ref = True)

Compiler crash traceback from this point on:
  File "/home/claudiofreire/venv/cython/lib/python2.7/site-packages/Cython/Compiler/ExprNodes.py", line 2017, in check_identifier_kind
    if entry.is_type and entry.type.is_extension_type:
AttributeError: 'NoneType' object has no attribute 'is_type'

@klaussfreire
Copy link
Contributor Author

Real-world code can be found here

@klaussfreire
Copy link
Contributor Author

This still happens in 0.28.3

@klaussfreire
Copy link
Contributor Author

Create #2317 which seems to fix this

@jakirkham
Copy link
Contributor

Should this be closed?

@klaussfreire
Copy link
Contributor Author

Yes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants