Iterator examples without `Clone`

[davidhewitt]

As originally posted on Gitter by @microgravitas:

Hi y'all! Can anyone point me to an example of how to implement an Python iterator which does not copy the whole underlying datastructure? All examples I could find used .clone(), something I'd like to avoid---if possible.

I've actually noticed this too about our iterator examples - it would be nice if we could find a way to make the .clone() unnecessary!

[birkenfeld]

There is obviously no way when using vec::IntoIter, but that is also not really necessary.

The iterator type should just need a reference to the Py object, and an index.

[davidhewitt]

Pretty much. I was thinking similar to the Python standard library types we might also want a guard to check that the base object is not modified during iteration. I think that'd have to be built into the base object, maybe as an additional field.

[rapodaca]

To be clear, is this issue about:

• using a Python Iterator from Rust, or
• using a Rust Iterator from Python?

If this issue isn't about using Rust Iterator from Python, what would be the best way to do this? In other words, can anyone answer this question?

From what I can tell, the problem is that iterator types returning references require a lifetime. And that lifetime propagates to any type that holds it. In PyO3, it's not possible for a struct to use a lifetime, as noted in this issue.

[davidhewitt]

This is exactly about using a "Rust iterator from Python". The example I'm thinking of is this one in the guide, which uses .clone().into_iter(), which is essentially just as bad as the stackoverflow link you post.

[rapodaca]

Yes, I've seen that one from the guide - thanks for the confirmation.

Is the problem that such a Rust Iterator to be used from Python (without clone) isn't possible, or that it's just not documented currently?

If it is possible, can you point me in the general direction of how to do it?

[davidhewitt]

It should be possible! From the discussion between myself and birkenfeld above, I think this might be the pair of types that you'd want:

#[pyclass]
struct Container {
    vec: Vec<usize>,

    // if you want to reproduce Python's "X modified during iteration" behaviour, you'll probably need
    // a flag like this to:
    //  - set it false whenever iteration starts
    //  - set it true every time the vec is modified
    //  - check it every time Iter.__next__ is called
    has_been_modified: bool,
}

#[pyclass]
struct Iter {
    src: Py<Container>,
    index: usize
}

I've left the implementation of the protocols out - it should be similar to the example I linked; hopefully you can figure it out from the types I've written. Feel free to ask further questions if what I've just written isn't clear enough!

If you manage to get this working, would you be willing to provide a PR to update the linked example? 🙂

[microgravitas]

@davidhewitt That looks like a fine solution for indexable data structures like vectors, but is there any chance of doing this for arbitrary rust iterators? I bumped into exactly the issue @rapodaca mentioned: it seems impossible to avoid adding a lifetime to anything that holds a reference to an iterator.

[davidhewitt]

For the more general case - the answer is that it's really really hard. PyO3 might be able to one day be able to do this, but it'll require a lot of careful R&D first.

See #1088 and #1089 for some thoughts I've had on this topic this morning.

[rapodaca]

My use cases appear to be the tough ones: wrap an existing iterator; or build an iterator to a non-indexable data structure such as HashSet.

#1089 is interesting. If I'm reading it right the idea is to build a Python iterator from Rust.

What about PyIterator? I'm not sure if it's still part of the project, but the line: " this class includes a Python<'p> token so that PyIterator can implement the rust Iterator trait" sounds encouraging.

I also found this which contains an example that isn't present anymore.

The documentation here is quite confusing. It's hard to know if what I'm looking at is out of date or current. Or even if it goes in the right direction.

[davidhewitt]

PyIterator is for the reverse direction - it's equivalent to calling iter(foo) on a Python object so that you can then iterate its contents.

And it's still part of the current version of pyo3 😄 https://docs.rs/pyo3/0.11.1/pyo3/types/struct.PyIterator.html