AttributeError on pointer dereferencing in loop target

[nissarin]

I was trying to use a "cleaver trick" in my very first attempt to write a Cython module, the code which failed to compile boils down to something like this:

cdef int[1] arr
cdef int *ptr = arr

for ptr[0] in range(100):
    pass

While this form doesn't make sense in pure Python, in C it's an equivalent of using *ptr which is fine, so I was wondering if it would be possible to support it in Cython as well, at the very least it would be nice to have a more helpful error message.

[scoder]

It's not clear to me what this code is supposed to do. Could you explain?

[nissarin]

This is just an example of syntax which can be compiled and run, if you want a concrete example - I was trying to use it in asn.1 parser like this:

for oid[0] in range(0, 3):
    if oid[1] < 40:
        break
    else:
        oid[1] -= 40

The first time I tried to cythonize file containing this piece of code I got an AttributeError and I had to comment out various parts to find what was causing it. So it would be nice to improve that part (error reporting) but if this gets be supported it would be even better.

[scoder]

Sorry, but I still don't understand it. Your example assigns to array index 0 as loop variable, and then uses array index 1 inside of the loop. What is the intention here? Do you really want to avoid the single code line that it would take to write this:

for i in range(0, 3):
    oid[0] = i
    if oid[1] < 40:
        ...

[scoder]

I can reproduce the AttributeError, BTW:

  File "Cython/Compiler/Nodes.py", line 425, in generate_execution_code                                                                                                                                                           
    stat.generate_execution_code(code)                                                                                                                                                                                                                                         
  File "Cython/Compiler/Nodes.py", line 6462, in generate_execution_code                                                                                                                                                          
    self.target.result(),                                                                                                                                                                                                                                                      
  File "Cython/Compiler/ExprNodes.py", line 443, in result                                                                                                                                                                        
    return self.calculate_result_code()                                                                                                                                                                                                                                        
  File "Cython/Compiler/ExprNodes.py", line 3804, in calculate_result_code                                                                                                                                                        
    return index_code % (self.base.result(), self.index.result())                                                                                                                                                                                                              
  File "Cython/Compiler/ExprNodes.py", line 443, in result                                                                                                                                                                        
    return self.calculate_result_code()                                                                                                                                                                                                                                        
  File "Cython/Compiler/ExprNodes.py", line 1305, in calculate_result_code                                                                                                                                                        
    return self.result_code                                                                                                                                                                                                                                                    
AttributeError: 'IntNode' object has no attribute 'result_code'                                                                                                                                                                                                                

I was using this code:

cdef int[2] oid
for oid[0] in range(0, 3):
    if oid[1] < 40:
        break

[scoder]

Oh, and Python does allow this:

>>> l = [0,1]
>>> for l[0] in range(3): pass
>>> l
[2, 1]

This also compiles correctly in Cython, so this is really just a bug.

[robertwb]

We should probably allow any lvalue as the loop target, as in Python, and we should produce better errors. I don't, however, understand setting old[0] and testing old[1] unless there's more to this loop that meets the eye. (Perhaps the intended C code would help?)

[nissarin]

oid points to an array of integers (function output), however in order to save space over the wire (function input, variable length encoded), the first two sub IDs are encoded according to formula (x * 40) + y, where x is in range 0..2. So my decoder first parsed the whole byte string, dumping values into array starting at index 1, then "unpacked" the first two sub IDs. Obviously I could've used a bit of arithmetic or nested if's but I choose to be "creative".

So.. yes, scoder is right, I did that just to avoid using additional temporary variable, at least that is what I would have done in C code, where abusing pointers is morally acceptable behaviour.

Consider following example (which works as intended):

spam.pyx

#cython: language_level=3

cdef struct Buffer:
    unsigned char *data
    int pos
    int len

cdef int parse_int(Buffer *buf) except *:
    cdef int intlen = buf.data[buf.pos]
    cdef unsigned ret = 0

    if intlen > 4:
        raise ValueError

    if buf.pos + intlen > buf.len:
        raise IndexError

    buf.pos += 1
    for buf.pos in range(buf.pos, buf.pos + intlen):
        ret = (ret << 8) | buf.data[buf.pos]

    return <int>ret

def parse(data):
    cdef Buffer b
    
    b.data = data
    b.pos = 0
    b.len = len(data)

    if b.len < 1:
        raise ValueError

    print('Before call', b.pos)
    ret = parse_int(&b)
    print('After call', b.pos)
    
    return ret

test.py

import spam

a = 123456789
b = a.to_bytes(4, 'big')
c = len(b).to_bytes(1, 'big')
d = c + b

print(spam.parse(d))

Output:

python3.6 test.py 
Before call 0
After call 4
123456789

One might want to rewrite this function, remove struct and pass the pointers directly: cdef int parse_int(unsigned char *data, int *pos, int len) but this would fail on pos[0].

[scoder]

but this would fail on pos[0].

As I said, that's a bug. Thanks for reporting it and thanks for the clarifications.

... just to avoid using additional temporary variable
... One might want to rewrite this function ...

Don't forget that there's a C compiler cleaning up after you. Aliasing it something that compilers learned decades ago.