Broadcast Array Indexing

[jecs]

I have an array x::Array{ImmutableArray.Vector2}.

It would be cool to be able to broadcast indexing as is done with functions.
e.g.
x.[1],
which would return the first element of each Vector2 in my array.

This is currently achievable with getindex.(x,1) or in the case of a range of indices
getindex.(x,[range]).

[mbauman]

I know your request is much more general than your example, but in this very specific case I'd use first.(x).

This syntax has also been proposed as a "pointwise" indexing operation — that is, instead of vectorizing over the array being indexed, you vectorize over the indices (#2591). Personally, I think that's more in-line with vectorized function calls; your suggestion would seem surprising to me.

[andyferris]

getindex.(x, 1) works fine for this.

@mbauman if getindex.() works then I would suggest .[] should be equivalent. If you want .[] to mean something different, then that would be confusing, wouldn't it?

(Actually, destructuring arrays like this is a very common thing to want to do, in some fields. Sometimes I even want something like vector..field where vector::Vector{T} and T has a field named field, to extract that field from each element. Currently getindex.() is slow for vectors of tuples and getfield.() is always slow, since inference doesn't work it's magic...)

[mbauman]

It just seems complicated and surprising to me. By analogy to the f.(args...) operator, I'd expect that only the indices in A.[inds...] would be broadcast over. We don't broadcast over arrays of functions.

This gets even worse with assignment. What parts of A.[inds...] = B should be broadcast over? Does it also lower to setindex!.(A, B, inds...)? Then what happens with A.[inds...] += B?

[jecs]

If B is compatible with A.[inds...], then I fail to see the problem.

For example,
A.[inds] += B --> setindex!.(A, getindex.(A , inds)+B, inds)
works.

[stevengj]

This seems reasonable to me. I'm trying to think of whether there is some other reasonable use for A.[inds], but nothing comes to mind. #2591 seems less compelling to me.

[StefanKarpinski]

Is this really entirely at odds with #2591? I feel like there's a generalization that covers both ideas. cc @toivoh

[mbauman]

I think the difficulty stems from the fact that indices work element-wise into an array. That is, I typically think of collections of indices and one atomic source array. By broadcasting over both the source and the indices you're treating both parts as though they're both collections of things. This really only makes sense if the source is a collection of arrays. Personally, I find that a lot less useful.

It's easiest to think about this with concrete examples. Pointwise indexing makes it trivial to extract the diagonal from a square matrix with A.[1:end, 1:end]. If the matrix is included in the broadcast part, that should be a BoundsError since it attempts to index the scalar elements of the matrix by (2,2) and (3,3). You'd have to protect the source in a collection itself with (A,).[1:end, 1:end].

Finally, I must say it's much easier to explain the behavior if the source array isn't a part of the broadcast operation. You broadcast the indices to a common size and index into the source. This works for both indexing and assignment. While the other semantic is really easy to implement, I'm not sure I know how to explain it without resorting to getindex.(A, I...).

[andyferris]

Isn't #2591 doable (and perhaps clearer) with a getindex that accepts a function?

By which I mean something like

diagonals = mat[i -> (i,i), 1:end]

In fact what you really want is a Generator

diagonals = mat[i,i for i = 1:end]

Would there be parsing issues with typed comprehensions here? Currently we try to construct Array with this syntax, giving an error. @mbauman Is "pointwise" indexing is more complex in some way?

I agree with @mbauman that it does seem that getindex.(A, inds) is confusing if inds is not scalar. That's really a tough one to nut out. Nonetheless, if we're going to have .+ parse as +.() and so-on, but not the [] operator, it would be surprising, so the pay-off would have to be large (and something we couldn't get with just a Generator, say). Of course [] is confusing for millions of reasons - it is used for so many things and has special parsing rules - like where does end come from in getindex.(a, 1:end)?

[jecs]

@andyferris I think that a nice way to deal with ambiguity would be to overload y.[x] such that if x is an array getindex.(y,x) is called; otherwise the argument is considered a tuple or enclosed in an array and getindex.(y,[x]) is called. I don't know how end is handled, so I'll leave that case to the gurus.

The type of indexing suggested in #2591 seems like a whole new brand of indexing, which should have a unique notation, such as x{inds...}, if it is permissible. .[] has the connotation of broadcasting.

[StefanKarpinski]

I think I agree with @mbauman's assessment that broadcasting over the collection being indexed into is too confusing and not terribly common. The pointwise indexing operation described in #2591, otoh, seems fairly useful to me – I've certainly used ind2sub and sub2ind with linear indexing to accomplish this sort of thing before. These days we want to avoid linear indexing wherever possible, especially in generic code, so it would be good to have a syntax for this. The difference seems to be that this issue wants broadcasting getindex over the collection and the indices whereas #2591 describes broadcasting the operation of indexing over just the indices. Not sure if that clarifies anything to anyone else. I'm inclined to support a version of #2591 that dovetails with broadcasting. What about having A.[X...] mean ((x...)->A[x...]).(X...), and whatever the corresponding definition for A.[X...] = ... would be.

[mbauman]

Yeah, I think we're talking past each other with three different behaviors here. Correct me if I'm wrong:

• Broadcasting pointwise indexing operator #2591: A.[X...] => broadcast((x...)->A[x...], X...). This is also Numpy's advanced indexing.
• @jecs: A.[X...] => map(a->a[X...], A). This indexes into each element of A with the given indices.
• @andyferris: A.[X...] => broadcast(getindex, A, X...). Coupled broadcasting across both A and the indices.

I'm actually rather surprised that #2591 hasn't garnered more support, but perhaps you're right that folks have just been getting by with linear indexing. I don't think we need to bend over backwards to find a meaning for .[] if there's not an obvious one.

[jecs]

@mbauman @andyferris and I are referring to the same thing, namely the behavior that you assigned to him.

A.[1:N] --> getindex.(A, [1:N])

A.[1] --> getindex.(A, 1) or getindex.(A,[1])

A.[[1,2,3,4]] --> getindex.(A,[1,2,3,4])

I = ones(3,3); A.[cat(2,I,2I)] --> getindex.(A, cat(2,I,2I))

I want the singleton expansion that is offered with broadcast. map does not handle that.

[StefanKarpinski]

I'm all for #2591. It was more out of left field when it was proposed, but these syntaxes are perfectly analogous:

f.(X...) => broadcast((x...)->f(x...), X...) # aka broadcast(f, X...)
A.[X...] => broadcast((x...)->A[x...], X...)

When you write f.(x...), f can't be a collection of functions, so it would be bizarre if A.[x...] expected A to be a collection of collections to index into.