Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AttributeError on pointer dereferencing in loop target #1831

Closed
nissarin opened this issue Aug 19, 2017 · 8 comments
Closed

AttributeError on pointer dereferencing in loop target #1831

nissarin opened this issue Aug 19, 2017 · 8 comments

Comments

@nissarin
Copy link

I was trying to use a "cleaver trick" in my very first attempt to write a Cython module, the code which failed to compile boils down to something like this:

cdef int[1] arr
cdef int *ptr = arr

for ptr[0] in range(100):
    pass

While this form doesn't make sense in pure Python, in C it's an equivalent of using *ptr which is fine, so I was wondering if it would be possible to support it in Cython as well, at the very least it would be nice to have a more helpful error message.

@scoder
Copy link
Contributor

scoder commented Aug 19, 2017

It's not clear to me what this code is supposed to do. Could you explain?

@nissarin
Copy link
Author

This is just an example of syntax which can be compiled and run, if you want a concrete example - I was trying to use it in asn.1 parser like this:

for oid[0] in range(0, 3):
    if oid[1] < 40:
        break
    else:
        oid[1] -= 40

The first time I tried to cythonize file containing this piece of code I got an AttributeError and I had to comment out various parts to find what was causing it. So it would be nice to improve that part (error reporting) but if this gets be supported it would be even better.

@scoder
Copy link
Contributor

scoder commented Aug 20, 2017

Sorry, but I still don't understand it. Your example assigns to array index 0 as loop variable, and then uses array index 1 inside of the loop. What is the intention here? Do you really want to avoid the single code line that it would take to write this:

for i in range(0, 3):
    oid[0] = i
    if oid[1] < 40:
        ...

@scoder
Copy link
Contributor

scoder commented Aug 20, 2017

I can reproduce the AttributeError, BTW:

  File "Cython/Compiler/Nodes.py", line 425, in generate_execution_code                                                                                                                                                           
    stat.generate_execution_code(code)                                                                                                                                                                                                                                         
  File "Cython/Compiler/Nodes.py", line 6462, in generate_execution_code                                                                                                                                                          
    self.target.result(),                                                                                                                                                                                                                                                      
  File "Cython/Compiler/ExprNodes.py", line 443, in result                                                                                                                                                                        
    return self.calculate_result_code()                                                                                                                                                                                                                                        
  File "Cython/Compiler/ExprNodes.py", line 3804, in calculate_result_code                                                                                                                                                        
    return index_code % (self.base.result(), self.index.result())                                                                                                                                                                                                              
  File "Cython/Compiler/ExprNodes.py", line 443, in result                                                                                                                                                                        
    return self.calculate_result_code()                                                                                                                                                                                                                                        
  File "Cython/Compiler/ExprNodes.py", line 1305, in calculate_result_code                                                                                                                                                        
    return self.result_code                                                                                                                                                                                                                                                    
AttributeError: 'IntNode' object has no attribute 'result_code'                                                                                                                                                                                                                

I was using this code:

cdef int[2] oid
for oid[0] in range(0, 3):
    if oid[1] < 40:
        break

@scoder
Copy link
Contributor

scoder commented Aug 20, 2017

Oh, and Python does allow this:

>>> l = [0,1]
>>> for l[0] in range(3): pass
>>> l
[2, 1]

This also compiles correctly in Cython, so this is really just a bug.

@robertwb
Copy link
Contributor

We should probably allow any lvalue as the loop target, as in Python, and we should produce better errors. I don't, however, understand setting old[0] and testing old[1] unless there's more to this loop that meets the eye. (Perhaps the intended C code would help?)

@scoder scoder changed the title Support for pointer dereferencing in loops AttributeError on pointer dereferencing in loop target Aug 20, 2017
@nissarin
Copy link
Author

oid points to an array of integers (function output), however in order to save space over the wire (function input, variable length encoded), the first two sub IDs are encoded according to formula (x * 40) + y, where x is in range 0..2. So my decoder first parsed the whole byte string, dumping values into array starting at index 1, then "unpacked" the first two sub IDs. Obviously I could've used a bit of arithmetic or nested if's but I choose to be "creative".

So.. yes, scoder is right, I did that just to avoid using additional temporary variable, at least that is what I would have done in C code, where abusing pointers is morally acceptable behaviour.

Consider following example (which works as intended):

spam.pyx

#cython: language_level=3

cdef struct Buffer:
    unsigned char *data
    int pos
    int len

cdef int parse_int(Buffer *buf) except *:
    cdef int intlen = buf.data[buf.pos]
    cdef unsigned ret = 0

    if intlen > 4:
        raise ValueError

    if buf.pos + intlen > buf.len:
        raise IndexError

    buf.pos += 1
    for buf.pos in range(buf.pos, buf.pos + intlen):
        ret = (ret << 8) | buf.data[buf.pos]

    return <int>ret

def parse(data):
    cdef Buffer b
    
    b.data = data
    b.pos = 0
    b.len = len(data)

    if b.len < 1:
        raise ValueError

    print('Before call', b.pos)
    ret = parse_int(&b)
    print('After call', b.pos)
    
    return ret

test.py

import spam

a = 123456789
b = a.to_bytes(4, 'big')
c = len(b).to_bytes(1, 'big')
d = c + b

print(spam.parse(d))

Output:

python3.6 test.py 
Before call 0
After call 4
123456789

One might want to rewrite this function, remove struct and pass the pointers directly: cdef int parse_int(unsigned char *data, int *pos, int len) but this would fail on pos[0].

@scoder
Copy link
Contributor

scoder commented Aug 20, 2017

but this would fail on pos[0].

As I said, that's a bug. Thanks for reporting it and thanks for the clarifications.

... just to avoid using additional temporary variable
... One might want to rewrite this function ...

Don't forget that there's a C compiler cleaning up after you. Aliasing it something that compilers learned decades ago.

@scoder scoder closed this as completed in 5ccc9e4 Sep 2, 2017
@scoder scoder added this to the 0.27 milestone Sep 2, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants