Safe wrapper for NPyIter

[PTNobel]

I'm evaluating using rust-numpy for a project that will require extensive use of NPyIter as I have to re-implement a number of ufuncs and need access to the broadcasting implementation.

I'm currently using np.broadcast in Python code to implement this.

Would there be any interest in making a safe wrapper for NPyIter if I built one?

Does anyone have any advice or similar wrappers I could reference? This is my first experience with writing unsafe rust code.

[kngwyu]

Thank you!
It's interesting to support NpyIter and I'm willing to give advises 🙂

Does anyone have any advice or similar wrappers I could reference?

Since NpyIter is not a Python type, we have no example and it can be difficult 🤔
Maybe we have to wrap it with a lifetime like NpyIter<'py> but I'm not sure how to do so.

[PTNobel]

Thanks kngwyu! I'll let you know when I get started on this and I'll post questions here?

As for the lack of examples, well I guess that just makes my job more fun.

Maybe we have to wrap it with a lifetime like NpyIter<'py> but I'm not sure how to do so.

Is your idea regarding lifetimes that a RustNpyIter object needs to have a litetime longer then the lifetime of any array used to create it?

[kngwyu]

Is your idea regarding lifetimes that a RustNpyIter object needs to have a litetime longer then the lifetime of any array used to create it?

Yes, maybe our NpyIter looks like:

struct NpyIter<'py> {
    ptr: NonNull<npyffi::NpyIter>,
    py: Python<'py>,
}

[kngwyu]

Note that npyffi::NpyIter is an opaque type so we can only access its pointer.

[PTNobel]

Any advice on ergonomicly handling the following parameters of NpyIter_New and NpyIter_MultiNew?

npy_uint32 flags: this is just a bunch of constants you can or together to change the behaviour of NpyIter.

npy_intp nop, PyArrayObject** op,: This is for MultiNew; I'm not sure how to best handle this, one obvious approach is to take a Vector of PyArrays or something along those lines, but that doesn't feel very clean. Any sense of what's standard/ergonomic for Rust?

[PTNobel]

Also entries in op can be NULL which causes the MultiIndex to allocate a new array.

[PTNobel]

The rust wrapper also needs to keep track of a seperate entry called the op_flags which are assoicated with each passed in op because that determines if we can return a &mut dtype or a &dtype. Also, is there a safe way to handle WRITEONLY pointers? Should we just initialize any value to 0 before returning it to avoid reading uninitialized memory in safe rust?

[PTNobel]

I'm thinking something inspired by OpenOptions from std::fs is probably best, as it should be easy and ergonimic to set the flags as desired. Not quite sure how best to specify the different options for the multiiter since some of them have to be seperated from flags and into op_flags.

I could use a different type for the MultiIter constructor and perhaps do something like constructor.set_flag(FlagEnum::FlagName).add_array(myfirst_pyarray).set_flag(OpFlagEnum::FlagName, true)..add_array(mysecond_pyarray).set_flag(OpFlagEnum::FlagName, true).set_flag(OpFlagEnum::AnotherFlagName, true). Note that constructor` and the return value of add_array would have to be different, but that probably makes sense anyways for this.

I also don't think there's a huge need for MultiIter and the 1D Iter to share a wrapper, as they do fundamentally behave differently.

[PTNobel]

I'm also thinking I should avoid NumPy setting any Exceptions, and instead have it return error strings that can be passed up to the caller so that the caller can choose whether to raise an exception.

[kngwyu]

something inspired by OpenOptions from std::fs

Maybe it is what is often called the builder pattern.

I'm also thinking I should avoid NumPy setting any Exceptions, and instead have it return error strings that can be passed up to the caller so that the caller can choose whether to raise an exception.

PyO3 itself can handle exceptions. You don't have to worry about it.

Also, is there a safe way to handle WRITEONLY pointers?

You can use MaybeUninit but it can be difficult.
For now, I recommend you to build the easiest one(maybe READONLY?) first.
Such safety problems are often hard so maintainers address them.

[messense]

I think we can close this issue?

[kngwyu]

👍🏼