Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: support contig iter op flag without buffering in nditer, or improve buffer usability #12433

Open
mattip opened this issue Nov 21, 2018 · 2 comments

Comments

@mattip
Copy link
Member

mattip commented Nov 21, 2018

nditer has a per-operand flag contig but it must be used with the global buffered flag:

>>> a = np.arange(6, dtype='i4')[::2]
>>> it = np.nditer(a, [], [['writeonly', 'contig']])
TypeError: Iterator operand required buffering, to be contiguous \
     as requested, but buffering is not enabled

The non-buffered contiguous case could be handled by either

  • Use writebackoncopy semantics to create a copy of the operand before iterating, and write back the results when the iterator is finished
  • At each iteration, create a local contiguous copy of the relevant data, and write it back at the beginning of each step. Write the final piece back when the iterator is finished.

The first method would be easier, but the second could allocate a smaller copy. This would be useful in, for instance, linalg and matmul ufuncs.

I admit I do not understand how the buffers work. Perhaps that could also provide what I am looking for, but the interface is not very well documented and the examples in the documentation and tests are not very helpful to me.

@mhvk
Copy link
Contributor

mhvk commented Nov 21, 2018

👍 But probably good to be even more explicit: currently, in linalg non-contiguous matrices are copied inside the gufunc implementations so that they can be passed on as contiguous blocks of data to blas. Might this be something the iterator could help take care of.

Just to show I (sadly) don't understand buffering either: with contig and buffering enabled, is copying/buffering still done if data is already contiguous? I'll add documentation to the labels since I also did not understand the documentation well...

@seberg
Copy link
Member

seberg commented Nov 27, 2018

Hmmm, should look into current buffering again before deciding on adding a second buffering mechanism next to it I guess. I don't remember anymore if the buffer was always used on all operands if active?
The buffer internally has some pretty big complexity because it tries to skip filling the buffer when it already holds the correct data for broadcasting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants