Adding support for saving matrices to files in npz format #154
Conversation
Codecov Report
@@ Coverage Diff @@
## master #154 +/- ##
==========================================
+ Coverage 96.89% 96.93% +0.04%
==========================================
Files 10 11 +1
Lines 1191 1207 +16
==========================================
+ Hits 1154 1170 +16
Misses 37 37
Continue to review full report at Codecov.
|
Overall, this is an awesome effort. We strive for perfection, so you can make changes and I'll approve. One thing not in this review is that you should add |
[0. , 0.86522495]]]) | ||
>>> os.remove('mat.npz') | ||
:param filename: string or file |
hameerabbasi
May 17, 2018
Collaborator
Parameters must be documented Numpy-style. See https://numpydoc.readthedocs.io/en/latest/format.html
Parameters must be documented Numpy-style. See https://numpydoc.readthedocs.io/en/latest/format.html
hameerabbasi
May 17, 2018
Collaborator
If you're using PyCharm, see "Docstring format" https://www.jetbrains.com/help/pycharm/settings-tools-python-integrated-tools.html
If you're using PyCharm, see "Docstring format" https://www.jetbrains.com/help/pycharm/settings-tools-python-integrated-tools.html
[0. , 0.86522495]]]) | ||
>>> os.remove('mat.npz') | ||
:param filename: file-like object, string, or pathlib.Path |
hameerabbasi
May 17, 2018
Collaborator
See comment above.
See comment above.
from sparse.utils import assert_eq | ||
|
||
|
||
def test_save_load_npz_file(): |
hameerabbasi
May 17, 2018
Collaborator
Try parametrizing compressed
as True/False
. See example parametrizations in test_coo.py
.
Try parametrizing compressed
as True/False
. See example parametrizations in test_coo.py
.
assert_eq(x, z) | ||
assert_eq(y, z.todense()) | ||
|
||
# test exception on wrong format |
hameerabbasi
May 17, 2018
Collaborator
Make this into another test.
Make this into another test.
""" | ||
|
||
nodes = {'data': matrix.data, | ||
'coords': matrix.coords} |
hameerabbasi
May 17, 2018
Collaborator
You want to store the shape too, sometimes these aren't enough on their own.
You want to store the shape too, sometimes these aren't enough on their own.
@@ -0,0 +1,96 @@ | |||
import numpy as np | |||
|
|||
from sparse.coo.core import COO |
hameerabbasi
May 17, 2018
Collaborator
This should be .coo.core
. Inside a package, we should always use relative imports.
This should be .coo.core
. Inside a package, we should always use relative imports.
@@ -0,0 +1,6 @@ | |||
io.load_npz |
hameerabbasi
May 17, 2018
Collaborator
Remove this file and build the docs. It will be auto-generated, then you can do git add
.
Remove this file and build the docs. It will be auto-generated, then you can do git add
.
@@ -0,0 +1,6 @@ | |||
io.save_npz |
hameerabbasi
May 17, 2018
Collaborator
Same as above.
Same as above.
with pytest.raises(RuntimeError): | ||
load_npz(filename) | ||
|
||
shutil.rmtree(dir_name) |
hameerabbasi
May 17, 2018
Collaborator
Oh, ignore. I just read the docs on that... It doesn't remove unless empty.
Oh, ignore. I just read the docs on that... It doesn't remove unless empty.
except KeyError: | ||
raise RuntimeError('The file {} does not contain a valid sparse matrix'.format(filename)) | ||
|
||
return COO(coords=coords, data=data) |
hameerabbasi
May 17, 2018
Collaborator
Add sorted=True, has_duplicates=False
and shape. The first two will make it a lot faster, and shape
ensures consistency.
Add sorted=True, has_duplicates=False
and shape. The first two will make it a lot faster, and shape
ensures consistency.
If this is too much for you, let me know and I'll fix it up. |
no worries, just got caught up at work. |
addressed all your comments and added the doc files |
Looks like most of the major concerns have been addressed, here are some other recommendations. Good work by the way. |
Whether to save in compressed or uncompressed mode | ||
Returns | ||
------- |
hameerabbasi
May 18, 2018
Collaborator
Remove this section, as the function isn't returning anything.
Remove this section, as the function isn't returning anything.
|
||
import sparse | ||
|
||
from sparse.io import save_npz, load_npz |
hameerabbasi
May 18, 2018
•
Collaborator
This still needs to be addressed. In scipy.sparse
, the functions are in scipy.sparse
, not scipy.sparse.io
.
Therefore, here, they should be in sparse
directly, not sparse.io
. You need to import them from sparse
here, not sparse.io
.
This still needs to be addressed. In scipy.sparse
, the functions are in scipy.sparse
, not scipy.sparse.io
.
Therefore, here, they should be in sparse
directly, not sparse.io
. You need to import them from sparse
here, not sparse.io
.
with pytest.raises(RuntimeError): | ||
load_npz(filename) | ||
|
||
shutil.rmtree(dir_name) |
hameerabbasi
May 18, 2018
Collaborator
flake8
is failing because of a missing newline here.
flake8
is failing because of a missing newline here.
[[0. , 0. ], | ||
[0. , 0.86522495]]]) | ||
>>> os.remove('mat.npz') | ||
hameerabbasi
May 18, 2018
•
Collaborator
Optional, but would be really nice to have: A see also section with load_npz
, np.savez
and scipy.sparse.save_npz
. Also a note saying that it's binary incompatible. See examples in our code and here: https://numpydoc.readthedocs.io/en/latest/format.html
Optional, but would be really nice to have: A see also section with load_npz
, np.savez
and scipy.sparse.save_npz
. Also a note saying that it's binary incompatible. See examples in our code and here: https://numpydoc.readthedocs.io/en/latest/format.html
[[0. , 0. ], | ||
[0. , 0.86522495]]]) | ||
>>> os.remove('mat.npz') | ||
hameerabbasi
May 18, 2018
•
Collaborator
Same as above:
Optional, but would be really nice to have: A see also section with save_npz
, np.load
and scipy.sparse.load_npz
. Also a note saying that it's binary incompatible.
Same as above:
Optional, but would be really nice to have: A see also section with save_npz
, np.load
and scipy.sparse.load_npz
. Also a note saying that it's binary incompatible.
@@ -53,3 +53,6 @@ API | |||
|
|||
where | |||
|
|||
save_npz |
hameerabbasi
May 18, 2018
Collaborator
Both of these should be added in alphabetical order, otherwise they're hard to find.
Both of these should be added in alphabetical order, otherwise they're hard to find.
A few more changes. We want the user-facing API to be compatible with |
>>> mat | ||
<COO: shape=(2, 2, 2), dtype=float64, nnz=2> | ||
>>> sparse.io.save_npz('mat.npz', mat) | ||
>>> loaded_mat = sparse.io.load_npz('mat.npz') |
hameerabbasi
May 18, 2018
Collaborator
Change this to sparse.load_npz
Change this to sparse.load_npz
>>> mat = sparse.COO(dense_mat) | ||
>>> mat | ||
<COO: shape=(2, 2, 2), dtype=float64, nnz=2> | ||
>>> sparse.io.save_npz('mat.npz', mat) |
hameerabbasi
May 18, 2018
Collaborator
Change this to sparse.save_npz
Change this to sparse.save_npz
>>> mat | ||
<COO: shape=(2, 2, 2), dtype=float64, nnz=2> | ||
>>> sparse.io.save_npz('mat.npz', mat) | ||
>>> loaded_mat = sparse.io.load_npz('mat.npz') |
hameerabbasi
May 18, 2018
Collaborator
Same comments as above. Also, it would be best to remove the examples section and in the main description, just add
See :obj:`save_npz` for usage examples.
Same comments as above. Also, it would be best to remove the examples section and in the main description, just add
See :obj:`save_npz` for usage examples.
This is in! Thanks, @nimroha! |
thank you! pleasure working with you
…On Sun, May 20, 2018, 16:45 Hameer Abbasi ***@***.***> wrote:
This is in! Thanks, @nimroha <https://github.com/nimroha>!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#154 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/APXkV-ORzdvI59Af740eEnn13VHn_BeNks5t0XOKgaJpZM4UDK5E>
.
|
Closes #153
100% test coverage python 2.7 and 3.5
added all the docs but could not build HTML locally