DOC: write doc about implicit behavior in advanced and basic combined indexing

[hidetatz]

Issue with current documentation:

It looks like there is an implicit behavior which is not documented in advanced and basic combined index.
Below is the reproducing code:

>>> x = np.arange(0, 12).reshape(2, 3, 2)
>>> y = x[0:2, [0, 1], 0:2]
>>> y.base
array([[[0, 1],
        [6, 7]],
                                                                                                                                                                                                                                                                                         
       [[2, 3],
        [8, 9]]])
>>> y.shape, y.strides
((2, 2, 2), (16, 32, 8))

y is created using advanced indexing.
Particularly, because the index [0:2, [0, 1], 0:2] contains both slice and list, this is called Combining advanced and basic indexing.

The point is because this is a kind of advanced indexing, this should return a copy of data. This is documented.

Advanced indexing always returns a copy of the data (contrast with basic slicing that returns a view).

However, in above example code, y is not a copy of data but view because it has non-None y.base .
I want when and how this behavior happens documented.

I want to contribute to this issue if possible, but I cannot because I also don't understand when and how this behavior happens.

Idea or request for content:

No response

[thalassemia]

Though I have no idea why y.base in your example is not None, I noticed that y.base.base is. Whatever array y is returning a view of, it is not the original array x. Modifying y has no effect on x and vice versa.

[bmwoodruff]

I'm in the process of reproducing this and noticed that I don't get the same output.

>>> x = np.arange(0, 12).reshape(2, 3, 2)
>>> y = x[0:2, [0, 1], 0:2]
>>> y.base
array([[[0, 1],
        [6, 7]],

       [[2, 3],
        [8, 9]]])
>>> y.shape, y.strides
((2, 2, 2), (8, 16, 4))

Note the difference in the last line, where your output was ((2, 2, 2), (16, 32, 8)).

[thalassemia]

Note the difference in the last line, where your output was ((2, 2, 2), (16, 32, 8)).

May I ask what OS, Python, and Numpy you are using? I got y.strides == (16, 32, 8) on both an M2 Macbook with Python 3.11.7 and Numpy 1.26.3 as well as an Ubuntu 22.04.4 x86 desktop with Python 3.12.0 and Numpy 2.0.0rc1.

Also, my initial post about y.base.base was misguided. The correct way of determining whether two arrays share memory is by calling np.shares_memory(x, y).

>>> x = np.arange(0, 12).reshape(2, 3, 2)
>>> y = x[0:2, [0, 1], 0:2]
>>> np.shares_memory(x, y) # Advanced indexing returns copy
False
>>> y = x[0:2, 0:2, 0:2]
>>> np.shares_memory(x, y) # Basic indexing returns view
True

For anyone curious, I did some digging and figured out why y.base is not None in this example. Turns out Numpy's main indexing function puts all advanced indices first and transposes them back if they happen to be consecutive. This transposition creates a new array object whose base is the non-transposed result.

This means that any combination of basic and consecutive advanced indices (where the first advanced index is for the second dimension or greater and needs to be transposed back) will yield an array whose base is not None.

>>> x = np.arange(2**4).reshape(2,2,2,2)
>>> x[:, [1], [1], [1]].base
array([[ 7, 15]])
>>> x[:, :, [1], [1]].base
array([[[ 3,  7],
        [11, 15]]])
>>> x[1, :, [1], :].base # Integer index treated as advanced index when combined so not consecutive
None
>>> x[:, 1, [1], :].base
array([[[ 6,  7],
        [14, 15]]])
>>> x[:, :, [1], :].base # Single advanced index counts as consecutive
array([[[[ 2,  3],
         [ 6,  7]],

        [[10, 11],
         [14, 15]]]])

[seberg]

The strides will be system dependent, just use floats or int64 explicitly (but not floats with arange).

However, in above example code, y is not a copy of data but view because it has non-None y.base .

No, a copy was made (check with np.may_share_memory). What is returned is simply a view into that copy (because internally NumPy does a transpose).

Going to close the issue for now, if you have a good idea on clarifying the docs that would be fine, but I don't think we should add too much there (since it might be more confusing than helpful).

[bmwoodruff]

May I ask what OS, Python, and Numpy you are using?

This was on Windows 11 running Python 3.12.1 with NumPy 1.26.3. On my Ubuntu Linux machine, I have the same output as you (Python 3.11.9 with Numpy 1.26.4). The Windows version seems to be the one that is different. Not sure this suggests a different issue to create.

[seberg]

@bmwoodruff no new issue, it is simply expected. I actually changed it for NumPy 2: The default integer on 64bit windows is now 64bit aligning it for all 64bit platforms. Let's see how that goes in practice...