Full array minus spares vector gives sparse array as output

[pmcvay]

A full array minus a sparse vector gives a sparse array with every position filled as an output. This seems inefficient. An example is below

julia> a = sparsevec([1,3,10], [.1, .3, .4])
10-element SparseVector{Float64,Int64} with 3 stored entries:
  [1 ]  =  0.1
  [3 ]  =  0.3
  [10]  =  0.4

julia> b = collect(1:10)
10-element Array{Int64,1}:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10

julia> b - a
10-element SparseVector{Float64,Int64} with 10 stored entries:
  [1 ]  =  0.9
  [2 ]  =  2.0
  [3 ]  =  2.7
  [4 ]  =  4.0
  [5 ]  =  5.0
  [6 ]  =  6.0
  [7 ]  =  7.0
  [8 ]  =  8.0
  [9 ]  =  9.0
  [10]  =  9.6

[pmcvay]

Here is a link to the discourse conversation on the matter https://discourse.julialang.org/t/why-does-julia-promote-full-array-to-sparse-array/44205

[dkarrasch]

@mbauman I did some digging, and this is what I found. It's this line

https://github.com/JuliaLang/julia/blob/68c71f577275a16fffb743b2058afdc2d635068f/stdlib/SparseArrays/src/higherorderfns.jl#L76

in which it is decided to materialize the broadcast in a sparse array (in the OP case). Is this intentional and meaningful in most/many other cases? Or could this be changed to return a dense array?

[mbauman]

It's doing what you'd want for multiplication.

We really need to re-work sparse broadcasts from scratch at some point. I think the ideal would do the same sort of thing that the structured LinearAlgebra types do: both statically and dynamically try to determine output type.

[dkarrasch]

Yes, I can see that for multiplication. But for generic operations, it feels like there is little reason to expect truly sparse output. Do you think one could redirect the computation at the top level somehow? Or make it generically return dense output, and rather special-case multiplication? May be considered as breaking, I guess...

[mbauman]

On the flip side, sparse folks tend to get more upset when we needlessly promote to dense as it can easily go OOM.

Some history: JuliaLang/julia#11474

[ViralBShah]

Yes, we generally want operations on sparse matrices to produce sparse results or the reason that @mbauman mentioned above. It is much clearer and readable to promote and leads to predictable performance.