Use size_hint in PyArray::from_iter.

[de-vri-es]

This PR changes the way the initial capacity is calculated for PyArray::from_iterator.

If size_hint gives an upper bound, that upper bound is used as the initial capacity. Else, either the minimum bound from the size hint or the original 512 / size_of::<T>() is used, whichever is larger.

Additionally, the PR fixes an off-by-one in the resize condition. The old check would resize the array one element too soon. That meant doubling the array size unnecessarily, even if the size hint gave a correct upper bound .

With these changes, from_iterator is now pretty much equally performant as from_exact_iter. The only remaining difference is that the loop still has to check if the capacity is exceeded, since we can't rely on the size hint for memory safety. Otherwise I would have considered deprecating that function entirely.

[kngwyu]

👍

Noticed what to be fixed

[kngwyu]

Thank you for your PR.
I don't know much about how size_hint is used, but is there no case where max_len is really large(e.g., (usize::MAX, Some(10)), which is problematic?

Additionally, the PR fixes an off-by-one in the resize condition. The old check would resize the array one element too soon. That meant doubling the array size unnecessarily, even if the size hint gave a correct upper bound .

Hoops, thank you for the fix.

[de-vri-es]

I don't know much about how size_hint is used, but is there no case where max_len is really large(e.g., (usize::MAX, Some(10)), which is problematic?

Theoretically, size_hint can return anything. However, an incorrect large upper or lower bound would be a buggy implementation. size_hint is intended exactly for reserving capacity. From the standard library documentation:

size_hint() is primarily intended to be used for optimizations such as reserving space for the elements of the iterator, but must not be trusted to e.g., omit bounds checks in unsafe code. An incorrect implementation of size_hint() should not lead to memory safety violations.

The default implementation of Iterator::size_hint returns (0, None) which still results in the old initial capacity. If an implementation does return incorrect large bounds, I would consider that a bug for that Iterator, not something we should protect against here.

[kngwyu]

That makes sense, thank you for explanation 👍

[de-vri-es]

Thanks for the review :)