-
-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RFC: SparseVectors, Take 2 #13440
RFC: SparseVectors, Take 2 #13440
Conversation
Thanks a lot for working on this! |
end | ||
half_screen_rows = limit ? div(rows - 8, 2) : typemax(Int) | ||
pad = ndigits(n) | ||
k = 0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this, and the k += 1
below are redundant since k
is the loop variable
How about deprecating |
Generally, I think I think we probably don't need |
I deleted comments that were completely off-topic here. |
doc""" | ||
ldltfact(::Union{SparseMatrixCSC,Symmetric{Float64,SparseMatrixCSC{Flaot64,SuiteSparse_long}},Hermitian{Complex{Float64},SparseMatrixCSC{Complex{Float64},SuiteSparse_long}}}; shift=0, perm=Int[]) -> CHOLMOD.Factor | ||
|
||
Compute the `LDLt` factorization of a sparse symmetric or Hermitian matrix. A fill-reducing permutation is used. `F = ldltfact(A)` is most frequently used to solve systems of equations `A*x = b` with `F\b`, but also the methods `diag`, `det`, `logdet` are defined for `F`. You can also extract individual factors from `F`, using `F[:L]`. However, since pivoting is on by default, the factorization is internally represented as `A == P'*L*D*L'*P` with a permutation matrix `P`; using just `L` without accounting for `P` will give incorrect answers. To include the effects of permutation, it's typically preferable to extact "combined" factors like `PtL = F[:PtL]` (the equivalent of `P'*L`) and `LtP = F[:UP]` (the equivalent of `L'*P`). The complete list of supported factors is `:L, :PtL, :D, :UP, :U, :LD, :DU, :PtLD, :DUP`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wording here is a little awkward - comma splice. Maybe F = ldltfact(A)
is most frequently used to solve systems of equations A*x = b
with F\b
. The factorization also the methods diag
, det
, logdet
defined.`?
Would be nice to have an explanation of what all these factors are (what is DUP
?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what is
DUP
I agree, but I'm not the one to write that documentation. I just moved the docs here from helpdb.jl.
I think for now it can be split into a combination of the orthogonal names |
r += 1 | ||
i2 = I[r] | ||
if i2 == i # accumulate r-th to the l-th entry | ||
V[l] = call(combine, V[l], V[r]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
combine(V[l], V[r])
to be similar with the SparseMatrix
construct?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, yup, thanks, this comes from SparseVectors.jl's 0.3 support.
So close. There'd be a clash of meanings with Is |
@@ -0,0 +1,1441 @@ | |||
### common.jl |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doesn't this need the standard license header? Also the file is not common.jl.
I'd asked before about transposing a sparse vector (but Viral deleted question), why couldn't that be returned as a sparse matrix? It wouldn't be quite as compact, but I'd think it would still be smaller than a dense array, or wouldn't it? |
Our sparse matrices are CSC format. It'd be good for you to thoroughly understand how CSC works if you're interesting in contributing to our sparse library. julia> S = sparse([1,10^7],[1,1],[1,1])
10000000x1 sparse matrix with 2 Int64 entries:
[1 , 1] = 1
[10000000, 1] = 1
julia> Base.summarysize(S)
88
julia> Base.summarysize(S')
80000080
julia> Base.summarysize(zeros(10^7))
80000000 |
I do understand perfectly well how the CSC format works. |
@@ -0,0 +1,649 @@ | |||
## sparsevec.jl |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This also has incorrect name and is missing Julia license info.
Looks really great! |
I see that you replaced the comparison |
The SparseMatrixCSC code is a little mixed as to whether it compares to |
I think we should be more explicit as to what the interface is assumed rather than having partially implemented / ill defined behavior. ex: julia> A = sparse([1,5,10],[1,5,10],['a','b','c'])
10x10 sparse matrix with 3 Char entries:
[1 , 1] = 'a'
[5 , 5] = 'b'
[10, 10] = 'c'
julia> full(A)
ERROR: MethodError: `zero` has no method matching zero(::Type{Char})
in full at sparse/sparsematrix.jl:227
julia> A[1]
'a'
julia> A[2]
ERROR: MethodError: `zero` has no method matching zero(::Type{Char})
in getindex at sparse/sparsematrix.jl:1398
in getindex at abstractarray.jl:483 but that might be another issue / PR. |
Thank you all for the review comments so far. Unless anyone has anything else, this is now ready as far as I'm concerned. I'm sure there will be a long tail of missing methods, performance improvements, and perhaps some bugs, but I think it's really quite good. It's well-covered by tests, and behaves as I'd expect. |
Renamings: * Rename sparse module to SparseArrays, and improve its separation from base. This makes it very simple to dynamically reload the sparse module. Move docstrings to their proper place * _copy_convert → collect * Rename sparsevector to the existing spzeros and sparsevec. * Use call overloading instead of call Remove functionality from SparseVectors.jl: * Simplify and remove some functors * Remove SparseVectorView * Remove no-longer-needed ambiguity preventers Add functionality for SparseVectors: * Add similar for SparseVector * Allow sparsevec(::AbstractArray), not just vectors * Add spzeros(n); adapt some tests to SparseVector * Allow CHOLMOD linalg with vectors * Implement (c)transpose(::SparseVector). Returns a dense vector since a one-row CSC structure is effectively dense but with worse performance. * Add vector sprandbool and allow passing RNG to all vector sprand* functions. Harden tests against random failures. * Implement, test and doc spones(::SparseVector) Improve performance for SparseVector indexing: * Punt to SparseMatrix for some indexing behaviors. Since the datastructures are compatible and SparseMatrix's routines are more optimized, it is easiest to just construct a temporary SparseMatrix and index into that. This is faster in all but the smallest of cases (N<10) * Use searchsorted for indexing SparseVector by UnitRange. This is about 20% slower on very small vectors, but is faster in general. Change SparseMatrix behaviors to take advantage of vectors * Add indexing behaviors for SparseMatrix->SparseVector * `vec` and `sparsevec` for CSC return SparseVectors * Update documentation to incorporate vectors Minor bugfixes and changes to SparseVectors: * Compare to literal 0 in vector construction and setindex. This matches SparseMatrix semantics, and makes indexing semantics consistent with regard to stored zeros * Use checkbounds more thoroughly * Use error types that are consistent with SparseMatrixCSC * Minor sparse vector display tweaks. Turn on output limiting by default, remove unused variable `k`, implement and use Base.summary * Fix missing return, add test Add some tests: * Add a test and comment to ensure nobody else tries to share data between vectors and matrices * Ensure indexing is consistent between one-column sparse matrices and sparse vectors, with special attention to stored zeros.
Both 64 bit linux workers got hit by the OOM killer. |
🍰 |
Great! |
🔢 Is there a constructor to create an all-zero sparse matrix of a given size? My brief foray didn't reveal one. |
spzeros? |
yep, thanks. On Wed, Oct 14, 2015 at 4:54 AM, Stefan Karpinski notifications@github.com
|
Great, thanks a lot! |
This is amazing! Thanks everyone. Kicking the tires. |
This introduces the
SparseVector
to the standard library. It is the descendent in spirit of #11424, but the entirety of the code is imported from JuliaSparse/SparseVectors.jl, with credit to @lindahua as the primary author. I've simply been working on getting it to mesh with Base idioms and names.TODO:
transpose(::SparseVector)
. Probably has to just return a dense array for now.module SparseMatrix
no longer makes sense.matrix
needs to be changed tovector or matrix
.Determine how or if we should deprecateDecision: punt. The current API works fine for now. We can revisit this later.sparsevec
. It's a conglomeration of two orthogonal behaviors that already have independent names:sparse
andvec
. Unfortunately, thesparse
function has so many signatures defined for it there's no unambiguous way to tack on thesparsevec
functionality for a few of the methods. Feedback about desired APIs here would be useful. See this comment for specifics.DefineDecision: wait to see if this is needed for writing generic code.rowvals(::SparseVector)
? Effectively the same functionality is defined as the currently-non-exportednonzeroinds
.spones(::SparseVector)
sprandbool(::Integer, ::AbstractFloat)