Add scatter operations

[yuehhua]

I have generalized scatter operations to every dimensions, instead of excluding the first dimension.
Related to yuehhua/ScatterNNlib.jl#32

[mcabbott]

Could these easily be combined to something like scatter!(+, dst, indices, src)?

[yuehhua]

@mcabbott Sure! I have API like scatter!(:add, dst, indices, src) in my ScatterNNlib.jl. Should I move it here as well?

[CarloLucibello]

I have API like scatter!(:add, dst, indices, src) in my ScatterNNlib.jl. Should I move it here as well?

I think he is suggesting to directly pass the operator (not a symbol), the same way you would use it in

julia> reduce(+, ones(10))
10.0

We could have

julia> function scatter!(op, ys::Array{T}, us::Array{T}, xs::Array{<:IntOrTuple}) where {T<:Real}
           @simd for k = 1:length(xs)
               k = CartesianIndices(xs)[k]
               ys_v = view(ys, xs[k]...)
               us_v = view(us, k)
               @inbounds ys_v .= (op).(ys_v, us_v)
           end
           ys
       end

and a few specialized methods:

function scatter!(op::typeof(mean), ys::Array{T}, us::Array{T}, xs::Array{<:IntOrTuple}) where {T<:Real}
...

[chengchingwen]

Maybe we should specified how our scatter/gather different from TF or Torch?

[yuehhua]

I discussed with @chengchingwen yesterday. We hope to introduce some new features to generalize scatter/gather operations. New features are listed in the following:

• Add dims argument for specifying which dimension to start scattering and the dimensions will be contiguous block, like TensorFlow does.
• The operational unit of scatter/gather could be a scalar or (at least) a vector. Both of them will be provided. The scalar unit will be designed by assigning dims=0, while the vector unit will be assigning with integer dims=1.
• Following above, dims=1 will be a default option.

• Scalar version: dst[i, j, k] = src[idx[i, j, k]...]
• Array version:
  • dims=1: dst[i, j, k] = src[:, idx[i, j, k]...]
  • dims=2: dst[i, j, k] = src[:, :, idx[i, j, k]...]
  • dims=3: dst[i, j, k] = src[:, :, :, idx[i, j, k]...]
  • ...

[CarloLucibello]

I wonder if typemax is the right thing to do here. The problem is if there are positions in dst which receive no contributions for src they will end up holding typemax, which doesn't seem meaningful. Maybe we should error out in such cases, but doing this check may have a performance impact

[yuehhua]

Yeah, I thought this issue before. Checking the position of dst is properly covered by idx is the way to avoid holding typemax. But still, it is necessary to check values in src is smaller than the value we assigned, either typemax or similar. similar gives the value existing in bare memory, so we have no idea knowing if the values are smaller enough.

[yuehhua]

What if we give the maximum of src? Thus, the value is at least smaller or equals to the maximum of src.

[mcabbott]

I think maximum(src) would be more surprising, in un-visited entries. typemax seems OK to me.

[yuehhua]

If it is ready to go, just let it go.

[CarloLucibello]

ok, I'll merge this so that work can proceed, I'll open a pr to revisit the docstrings