Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Sparsity-preserving outer products #24980
This PR adds sparsity-preserving outer products of sparse vectors and views into sparse matrices, implemented as methods of
The motivation that caused me to dig into this is the desire to have fast/efficient quadratic matrix products. My specific case is to compute a very large sparse-dense-sparse product, but computation of the middle dense matrix can be parallelized as computations of single columns.
Some example code showing the impact:
using BenchmarkTools using Test m,n = (25_000, 100_000); # dimensions of matrices k = 1000; # column vector location to fill A = sprand(m, n, 0.01); # large, sparse operator b = rand(n); # column subset of dense matrix B = sparse(collect(1:n), fill(k, n), b, n, n); # sparse representation if true # outer products only @btime $b * $A[:, $k]'; @btime $b * view($A, :, $k)'; # quadratic matrix products which motivated C1 = @btime $A * $B * $A'; C2 = @btime ($A * $b) * view($A, :, $k)'; @test C1 == C2 end
On master (Version 0.7.0-DEV.2770, Commit 11d5a53):
I've marked this as a work-in-progress because I'd welcome comments on how to potentially better integrate the new methods with existing code. The first commit could be thrown away / extracted to separate PR — it was just something I noticed while grapping with current code —
2 times, most recently
Dec 10, 2017
I have been meaning to return to this for quite some time, so yes, but possibly not until mid-January or so. Between the holidays and my own academic work, I don't have much free time in the next few weeks. (Maybe I'll squeeze it in between things as my own form of relaxing and getting away from Matlab ;-P)
As a reference point for myself — current master (v1.2.0-DEV.27) using similar benchmarks as the original post destroys my computer for a few minutes. I think the second
I've reimplemented this PR on the current master. The big differences from before are:
The updated benchmark I'm working with is:
using BenchmarkTools, SparseArrays, Test m,n = (2_000, 10_000); k = 1000; A = sprand(m, n, 0.01); a = view(A, :, k); u = A[:, k]; x = sparse(rand(n)); X = sparse(collect(1:n), fill(k, n), x, n, n); suite = BenchmarkGroup() # outer products only suite["x × u'"] = @benchmarkable $x * $(u'); suite["x × a'"] = @benchmarkable $x * $(a'); # quadratic matrix products which motivated suite["A × X × A'"] = @benchmarkable $A * $X * $(A'); suite["(A × x) × a'"] = @benchmarkable ($A * $x) * $(a'); tune!(suite) before = run(suite) using Revise Revise.track(Base) after = run(suite) if true println("Before vs after for specific operations") println("=======================================\n") for key in keys(after) t1, t0 = minimum.((after[key], before[key])) r = judge(t1, t0) print(key, ": ") show(stdout, MIME"text/plain"(), r) println() end println("\n\n") println("Optimized quadratic product, before and after") println("=============================================\n") for (trial,name) in [(before,"before"), (after,"after")] key1 = "(A × x) × a'" key2 = "A × X × A'" t1, t0 = minimum.((trial[key1], trial[key2])) r = judge(t1, t0) print("$key1 vs $key2 ($name): ") show(stdout, MIME"text/plain"(), r) println() end end
and gives results
The outer product part is straightforward. If @mbauman or @KristofferC can check the broadcasting bits, we can probably merge. @andreasnoack mentioned about
Also, it is fine to get this performance improvement in and do further improvements in other PRs.
mbauman left a comment
I'm impressed — the sparse broadcasting machinery has survived a quite a few incremental code changes and is a bit of a tangled mess. You've adeptly hooked in right where it'd make sense.
I think doing this kind of peep-hole optimization is worth doing and with a few minor renames we can set the stage for supporting more and more array types without needing to
Correct. The old PR had put in a stub function, but any performance "improvement" implied by providing an inplace update would have been a lie since I wasn't implementing such a feature at that time. If someone wants to tackle that, I agree that it can be in a new PR.
Thanks!! I did have to abandon several approaches after realizing it wasn't going to work out as I'd hoped.
@ViralBShah has the current story correct — since I started this PR a year ago, the switch to
After the rewrite to current master, I dropped one of my original goals — having dense-sparse outer products produce a sparse output. I decided I was unsure enough of the broadcasting implementation at this point to spend time working on that goal; issue #26613 would be a good place to revive that conversation and let those enhancements occur in synchrony with a future PR to extend this implementation.
Note that CI runs on the merge commit so rebasing doesn't really do anything w.r.t to CI.
I don't really see how this can be backported to a patch release since it is changing behavior? Tentatively removing backport label. Also, this needs a NEWS entry.