Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Advancing iterator in range loop invalidates previously dereferenced values for input iterators in C++ #3055

ozars opened this issue Jul 26, 2019 · 1 comment


Copy link

commented Jul 26, 2019

I have a user-defined class implementing C++ input iterator requirements. It skips an element in the beginning and prints last element twice when I iterate over it with Cython's range based loop. Looking into produced code, I realized iterator value is increased before the loop body gets executed. Incrementing is done right after dereferenced iterator value is copied into a temporary. However, incrementing the iterator invalidates previously dereferenced values for input iterators. In my case, it was trying to parse next node in a stream.

Here is a toy example for demonstration:

# distutils: language = c++

from cython.operator cimport dereference as deref, preincrement as preinc

cdef extern from *:
    //#define PRINT() std::cout << __PRETTY_FUNCTION__ << std::endl
    #define PRINT()
    struct CountDown {
        struct Iterator {
            CountDown* ptr;
            Iterator() = default;
            Iterator(CountDown* ptr) : ptr(ptr) {}
            Iterator& operator++() { PRINT(); ptr->count--; return *this; }
            Iterator& operator++(int) { PRINT(); ptr->count--; return *this; }
            const int* operator*() { return &ptr->count; }
            bool operator!=(const Iterator&) { PRINT(); return ptr->count > 0; }
        int count;
        CountDown() = default;
        CountDown(int count) : count(count) {}
        Iterator begin() { PRINT(); return Iterator(this); }
        Iterator end() { PRINT(); return Iterator(); }
    cdef cppclass CountDown:
        cppclass Iterator:
            Iterator operator++()
            Iterator operator++(int)
            const int* operator*()
            bint operator!=(Iterator)
        CountDown(int count)
        Iterator begin()
        Iterator end()

cdef countdown_range():
    cdef CountDown cd = CountDown(5)
    cdef const int* num
    for num in cd:

cdef countdown_expected():
    cdef CountDown cd = CountDown(5)
    cdef CountDown.Iterator it = cd.begin()
    while it != cd.end():
        it = preinc(it)

print("Actual output:")
print("Expected output:")


 ~/tmp/cyissue  python3 -c "import example"
Actual output:
Expected output:

Here is the related part in the produced code:

  /* "example.pyx":42
 *     cdef CountDown cd = CountDown(5)
 *     cdef const int* num
 *     for num in cd:             # <<<<<<<<<<<<<<
 *         print(deref(num))
  __pyx_t_1 = __pyx_v_cd.begin();
  for (;;) {
    if (!(__pyx_t_1 != __pyx_v_cd.end())) break;
    __pyx_t_2 = *__pyx_t_1;
    __pyx_v_num = __pyx_t_2;

This isn't a crucial feature since it can be implemented without range-based loop as in the above example, yet this was quite surprising for me and it took some time to figure this out, so I decided to open this issue.

@ozars ozars changed the title Advancing iterator before dereferencing invalidates input iterators in C++ Advancing iterator in range loop invalidates previously dereferenced values for input iterators in C++ Jul 26, 2019


This comment has been minimized.

Copy link

commented Jul 26, 2019

Below part is where this loop translation happens AFAICS:

def generate_execution_code(self, code):
old_loop_labels = code.new_loop_labels()
code.putln("for (;;) {")
self.item.generate_evaluation_code(code), code)
break_label = code.break_label

This translates for s in seq: loop_body expression to something like:

it = iter(seq)
while True:
   s = next(it) or break loop

whereas correct interpretation for C++ should have been something like:

it = seq.begin()
while True:
   if it is seq.end() break loop
   s = *it

So... It looks like this issue is caused by a subtle difference between semantics of python's next and C++'s iterators. In python next does what operator++ and operator* together do in C++. I'm not sure how this can be fixed (splitting NextNode into two pieces for C++? Perhaps introducing a ForNode to split C++ implementation altogether?). Given current implementation works fine for the vast majority of cases, it may not worth the effort though, but it may be useful to note this quirk somewhere in the documentation at least.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
1 participant
You can’t perform that action at this time.