Repository navigation
RFC: sparse: establish an official policy for duplicate elements #5807
Description
Activity
+10 for documenting this kind of thing clearly.
(1) and (2) of the proposal seem fine to me, I'm not sure about (3). I was trying to find a place in our docs where we explain the structure of the matrix formats and if/how to manually modify data. But there's very little in the tutorial or refguide - the best description is probably this one: http://www.scipy-lectures.org/advanced/scipy_sparse/index.html
This is blocking: (at least) gh-5394
Agreed on that it would have been better to not permit duplicates. I understand they are useful at matrix assembly phase, but I think this would be better handled by a specialized matrix format rather than forcing it everywhere. It is unclear to me if the present support for duplicates has good uses; I guess most either use lil/dok (which are also inefficient due to Python overhead), or accumulate coo-format arrays manually.
I would go allowing canonicalization at any time (1).
Adding a flag that user is suppose to toggle is just asking for subtle broken programs. +1 to canonicalization at any time (1).
Yeah, it seems like (3) could be better handled by adding some documentation, rather than more complexity in the code.
It looks like (1) is agreed to be a good choice. We should figure out a good place to document it, then inform the mailing list.
I think we can implement (2) pretty cheaply by adding a
warn=Falsekeyword argument tosum_duplicates(), and force internal callers to passwarn=True. It's not a particularly elegant solution, but I like it better than usinginspectto peek at the caller's stack frame.I first used sparse matrices in MATLAB for finite element problems (in the 1990s). Coming from that background I regard the summation of duplicates as a useful, if not essential, feature. I just used it in a Stackoverflow answer, http://stackoverflow.com/a/37040831/901925
The only documentation that I'm aware of is on the
coo_matrixpageBy default when converting to CSR or CSC format, duplicate (i,j) entries will be summed together. This facilitates efficient construction of finite element matrices and the like. (see example)
I haven't studied the compile code much, so don't know exactly where and how the summation is implemented. My observation is that the
.rowand.colattributes of acooformat might actually be the arrays provided as input in the(data, (i, j))pattern (if they are arrays of the correct dtype). So at least in some cases duplicates are present simply because inputs have not been processed or checked.I haven't observed whether duplicates are allowed with the other formats.
bmatusescoomatrices as intermediates, but doesn't allow overlapping blocks. It is obvious how alilcould have duplicates.dokis a dictionary, so it can't have duplicates, can it?I haven't taken time yet to look at the cited issues where this seems to be a problem. But I just wanted to express support for the existing practice. Documentation can always be improved without affecting legacy practices.
Hi all,
I'd like to contribute a big -1 to this discussion. #5394 already sent me on a bug hunt and reduced efficiency in my own code (since I have to explicitly make a copy of a
coo_matrixbefore converting to CSR). As mentioned in #6603, in that PR, canonization occurs even whencopy=Trueis passed totocsr().One of the motivating examples is #4409, which in my reading points to an application (csgraph) that absolutely cannot work when sparse is canonizing arrays left, right, and center, namely, the existence of graph edges with weight 0 (as opposed to non-edges).
Another, #5806, handled the non-canonical case cleanly.
I can't count right now the number of places I use the internal structure of COO and CSR. It's certainly used within scikit-image. In my own code I've implemented a CSR matrix that can get expanded with new rows (here). I'd expect arbitrary canonization to wreak havoc with it. Along with any number of user subclasses.
My suggestion is not to mess with users' objects unless explicitly allowed by the user. For example, you could add
canonize=Falseto all the places where canonization would be useful. IfFalse, make a copy of the original array if necessary. This mirrors the common pattern of having anoutput=inputkwarg for in-place operations.In short, this isn't a small case of "we might break some users' code but it's ok since it's undocumented". This has massive downstream ramifications.
To be clear, my argument is -1 for the policy being in-place canonization by default, not for establishing a policy. =P
@jni: Note that "canonicalization" in this context refers only to duplicate entries, not explicit zeros. The discussion on explicit zeros (gh-3343) seems to have converged on preservation, specifically for the csgraph example you mentioned. If there are operations which don't preserve explicit zeros (other than, say,
eliminate_zeros()) those should be reported as bugs.That said, there is also a case for using duplicate entries in a sparse matrix to represent a multigraph. I personally think that this goes a little too far, and that a specialized data structure would be more appropriate. I'd like to hear what others think on the topic.
My ideal is to allow duplicate entries only during construction via
__init__: once thespmatrixsubclass is initialized, all of its internal members should be duplicate-free and unchanged by normal operations. This would allow us to get rid of the numerousself.has_canonical_formatchecks and calls tosum_duplicates()which currently litter the codebase, eliminating all the problems with in-place canonicalization. It does mean that the arguments passed to the__init__method may end up getting copied in some cases, but judicious use of kwargs should allow the user to say "I know what I'm doing, just use these arrays directly" with the understanding that duplicate entries will cause incorrect results for some operations.I'm interested to find out which use-cases would break under this "canonicalize-at-init" proposal. I can't think of any at the moment, but I'm sure some exist!
- Canonicalization on init may have the same issue, because then with the (default) copy=False, `__init__` would modify the provided arrays. With copy=True, the data arrays could contain different data, which also may break assumed invariants. . The question here is deciding where the line of "spacebar heating" goes. Strict backward compatibility doesn't really allow canonicalization at any stage AFAICS so the internal format would be stuck with duplicates etc., with the costs what that entails.
- added a commit that references this issue
on Sep 28, 2016 @pv My attempt is linked above. If you pass
copy=False, canonicalize=False, then no checking is done and the arrays are used unmodified.The default as of now is
copy=False, canonicalize=True, which is slightly counterintuitive because it does make a copy in the case where canonicalization is actually needed.Perhaps something like
copy=Nonecould be the default, indicating that we'll try to avoid copying if possible, but don't guarantee it. In that case, a warning would be raised if the user asks for canonicalization withcopy=Falsedirectly.- added 2 commits that reference this issue
on Sep 29, 2016 If you pass
copy=False, canonicalize=False, then no checking is done and the arrays are used unmodified.... And everything you do with that matrix from then onwards is potentially broken. This isn't a very reassuring statement.
The default as of now is
copy=False, canonicalize=TrueHow so? In 0.18, if I create a COO matrix, thankfully, the
row,column, anddataarrays are used untouched.I'm interested to find out which use-cases would break under this "canonicalize-at-init" proposal
- I have a row-expandable CSR class that subclasses
csr_matrix, here. It works by replacingdata,indices, andindptrwith properties that return views into a larger array (and this larger array gets reallocated if needed). I suspect that it depends quite strongly on the array internals not being messed with. - I wasn't thinking of it in terms of a multi-graph, but that is actually in a strong sense what I am doing here (also used in scikit-image here). Note that both cases depend on the COO matrix not being canonicalized at build time. In the first case, the call
copy()in this line was made necessary by canonicalization duringtocsr()in 0.18, which, as noted in scipy.sparse.coo_matrix.tocsr modifies the row, col, and data attributes of the original coo_matrix #6429 and BUG (?) COO -> CSR/CSC unexpectedly modifies internal data #6603, is unexpected in a function that is intended to return a copy of the input array.
- I have a row-expandable CSR class that subclasses
It seems to me the only fully backward compatible behavior is to not canonicalize at any stage. Canonicalization could be done at construction time with copy=True, or when constructing new arrays in arithmetic operations etc. --- also that is in principle backward incompatible change, but could be more friendly towards code that relies on the undefined behavior vs modifying the internal state of the matrices.
I would like to contribute my use-case here, since this caused some recent issues after upgrading to 0.18. I've got some finite element code where I was using
coo_matrixprimarily as storage during the matrix assembly process. This assembly happens multiple times since I'm working with a non-linear problem that requires being solved iteratively. After assembly at each iteration, I convert thecoo_matrixto acsr_matrix, where duplicate entries are summed automatically. This sort of thing is encouraged by the documentation describing the intended usage ofcoo_matrix(as mentioned above)- COO is a fast format for constructing sparse matrices
- Once a matrix has been constructed, convert to CSR or CSC format for fast arithmetic and matrix vector operations
- By default when converting to CSR or CSC format, duplicate (i,j) entries will be summed together. This facilitates efficient construction of finite element matrices and the like. (see example)
The problem for me happens after the first iteration, because calling
.to_csc()(among other methods) unexpectedly causes elimination of duplicate entries in thecoo_matrix. This did not happen in versions < 0.18. When the assembly process happens for the second iteration, thecoo_matrixhas been modified, and there is a problem because some of the entries are now gone. My workaround was to simply store the matrix data in my own custom class (since I no longer have the needed control overcoo_matrix), and then convertMyCooMatrix -> coo_matrix -> csr_matrixat each iteration.The documentation above gives the impression that
coo_matrixshould be used for assembly, and my particular use case seems to illustrate a potential problem with the current behavior. The documentation also suggests that thecoo_matrixis more for convenience rather than performance, and thus one should expect to deal with potential inefficiency with the added convenience of having duplicates allowed.
Background
As an admission to performance and ease of construction, most of our sparse matrix formats admit duplicate entries. For example:
Unfortunately, this causes a lot of problems for sparse matrix operations (#4409, #5394, #5806), and causes confusion regarding the true meaning of
nnz, even ignoring the issue of explicit zeros (#3343).Most sparse matrix formats have a method
sum_duplicates()which operates in-place to canonicalize the internal storage, but it's unclear whether other methods are allowed to call this without first making a copy (see #5741 (comment)).Proposal
I think that allowing duplicate entries in internal sparse matrix representations was an API mistake, but now that we allow it, it's difficult to disallow without breaking lots of existing user code. Therefore, I propose that we make it a policy that:
sum_duplicates()triggers in-place canonicalization, aSparseEfficiencyWarningis thrown, to alert the user that something potentially unexpected is going on.If this gets traction here, I'll send it along to the Scipy-dev mailing list as well. Thoughts?