Adding support for saving matrices to files in npz format #154

nimroha · May 17, 2018

Closes #153

100% test coverage python 2.7 and 3.5

added all the docs but could not build HTML locally

codecov-io · May 17, 2018

Codecov Report

Merging #154 into master will increase coverage by 0.04%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #154      +/-   ##
==========================================
+ Coverage   96.89%   96.93%   +0.04%     
==========================================
  Files          10       11       +1     
  Lines        1191     1207      +16     
==========================================
+ Hits         1154     1170      +16     
  Misses         37       37

Impacted Files	Coverage Δ
sparse/io.py	`100% <100%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f939529...d5c19a7. Read the comment docs.

hameerabbasi · May 17, 2018

hameerabbasi requested changes May 17, 2018

View changes

Overall, this is an awesome effort. We strive for perfection, so you can make changes and I'll approve. 😄

One thing not in this review is that you should add load_npz and save_npz into the list of functions in docs/generated/sparse.rst before building the docs.

hameerabbasi · May 17, 2018

sparse/io.py

+            [0.        , 0.86522495]]])
+    >>> os.remove('mat.npz')
+
+    :param filename: string or file


Parameters must be documented Numpy-style. See https://numpydoc.readthedocs.io/en/latest/format.html

If you're using PyCharm, see "Docstring format" https://www.jetbrains.com/help/pycharm/settings-tools-python-integrated-tools.html

hameerabbasi · May 17, 2018

sparse/io.py

+            [0.        , 0.86522495]]])
+    >>> os.remove('mat.npz')
+
+    :param filename: file-like object, string, or pathlib.Path


See comment above.

hameerabbasi · May 17, 2018

sparse/tests/test_io.py

+from sparse.utils import assert_eq
+
+
+def test_save_load_npz_file():


Try parametrizing compressed as True/False. See example parametrizations in test_coo.py.

hameerabbasi · May 17, 2018

sparse/tests/test_io.py

+    assert_eq(x, z)
+    assert_eq(y, z.todense())
+
+    # test exception on wrong format


Make this into another test.

hameerabbasi · May 17, 2018

sparse/io.py

+    """
+
+    nodes = {'data':   matrix.data,
+             'coords': matrix.coords}


You want to store the shape too, sometimes these aren't enough on their own.

hameerabbasi · May 17, 2018

sparse/io.py

@@ -0,0 +1,96 @@
+import numpy as np
+
+from sparse.coo.core import COO


This should be .coo.core. Inside a package, we should always use relative imports.

hameerabbasi · May 17, 2018

docs/generated/sparse.io.load_npz.rst

@@ -0,0 +1,6 @@
+io.load_npz


Remove this file and build the docs. It will be auto-generated, then you can do git add.

hameerabbasi · May 17, 2018

docs/generated/sparse.io.save_npz.rst

@@ -0,0 +1,6 @@
+io.save_npz


Same as above.

hameerabbasi · May 17, 2018

sparse/tests/test_io.py

+    with pytest.raises(RuntimeError):
+        load_npz(filename)
+
+    shutil.rmtree(dir_name)


Try os.rmdir

Oh, ignore. I just read the docs on that... It doesn't remove unless empty.

hameerabbasi · May 17, 2018

sparse/io.py

+        except KeyError:
+            raise RuntimeError('The file {} does not contain a valid sparse matrix'.format(filename))
+
+    return COO(coords=coords, data=data)


Add sorted=True, has_duplicates=False and shape. The first two will make it a lot faster, and shape ensures consistency.

hameerabbasi · May 17, 2018

If this is too much for you, let me know and I'll fix it up. 🙂 The PR will still be in your name.

nimroha · May 17, 2018

no worries, just got caught up at work.
will work on it tonight

nimroha · May 17, 2018

addressed all your comments and added the doc files

hameerabbasi · May 18, 2018

hameerabbasi requested changes May 18, 2018

View changes

Looks like most of the major concerns have been addressed, here are some other recommendations. Good work by the way. 😄

hameerabbasi · May 18, 2018

sparse/io.py

+        Whether to save in compressed or uncompressed mode
+
+    Returns
+    -------


Remove this section, as the function isn't returning anything.

hameerabbasi · May 18, 2018

sparse/tests/test_io.py

+
+import sparse
+
+from sparse.io import save_npz, load_npz


This still needs to be addressed. In scipy.sparse, the functions are in scipy.sparse, not scipy.sparse.io.

Therefore, here, they should be in sparse directly, not sparse.io. You need to import them from sparse here, not sparse.io.

hameerabbasi · May 18, 2018

sparse/tests/test_io.py

+    with pytest.raises(RuntimeError):
+        load_npz(filename)
+
+    shutil.rmtree(dir_name)


flake8 is failing because of a missing newline here.

hameerabbasi · May 18, 2018

sparse/io.py

+           [[0.        , 0.        ],
+            [0.        , 0.86522495]]])
+    >>> os.remove('mat.npz')
+


Optional, but would be really nice to have: A see also section with load_npz, np.savez and scipy.sparse.save_npz. Also a note saying that it's binary incompatible. See examples in our code and here: https://numpydoc.readthedocs.io/en/latest/format.html

hameerabbasi · May 18, 2018

sparse/io.py

+           [[0.        , 0.        ],
+            [0.        , 0.86522495]]])
+    >>> os.remove('mat.npz')
+


Same as above:

Optional, but would be really nice to have: A see also section with save_npz, np.load and scipy.sparse.load_npz. Also a note saying that it's binary incompatible.

hameerabbasi · May 18, 2018

docs/generated/sparse.rst

@@ -53,3 +53,6 @@ API

    where

+    save_npz


Both of these should be added in alphabetical order, otherwise they're hard to find.

hameerabbasi · May 18, 2018

hameerabbasi reviewed May 18, 2018

View changes

A few more changes.

We want the user-facing API to be compatible with scipy.sparse, so the .io should be removed everywhere and it should just be treated as an internal submodule.

hameerabbasi · May 18, 2018

sparse/io.py

+    >>> mat
+    <COO: shape=(2, 2, 2), dtype=float64, nnz=2>
+    >>> sparse.io.save_npz('mat.npz', mat)
+    >>> loaded_mat = sparse.io.load_npz('mat.npz')


Change this to sparse.load_npz

hameerabbasi · May 18, 2018

sparse/io.py

+    >>> mat = sparse.COO(dense_mat)
+    >>> mat
+    <COO: shape=(2, 2, 2), dtype=float64, nnz=2>
+    >>> sparse.io.save_npz('mat.npz', mat)


Change this to sparse.save_npz

hameerabbasi · May 18, 2018

sparse/io.py

+    >>> mat
+    <COO: shape=(2, 2, 2), dtype=float64, nnz=2>
+    >>> sparse.io.save_npz('mat.npz', mat)
+    >>> loaded_mat = sparse.io.load_npz('mat.npz')


Same comments as above. Also, it would be best to remove the examples section and in the main description, just add

See :obj:`save_npz` for usage examples.

hameerabbasi · May 20, 2018

hameerabbasi approved these changes May 20, 2018

View changes

hameerabbasi · May 20, 2018

This is in! Thanks, @nimroha!

nimroha · May 20, 2018

thank you! pleasure working with you

…

On Sun, May 20, 2018, 16:45 Hameer Abbasi ***@***.***> wrote: This is in! Thanks, @nimroha <https://github.com/nimroha>! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#154 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APXkV-ORzdvI59Af740eEnn13VHn_BeNks5t0XOKgaJpZM4UDK5E> .

nimroha force-pushed the nimroha:npz_file_IO branch from 58ecee6 to 770e433 May 17, 2018

added .rst files

Loading status checks…

34aec2c

fixed more PR comments

Loading status checks…

d5c19a7

pydata/sparse

Join GitHub today

Adding support for saving matrices to files in npz format #154

Conversation

nimroha commented May 17, 2018 • edited by hameerabbasi

This comment has been minimized.

codecov-io commented May 17, 2018 • edited

Codecov Report

hameerabbasi requested changes May 17, 2018 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hameerabbasi commented May 17, 2018

This comment has been minimized.

nimroha commented May 17, 2018

NimrodM and others added some commits May 17, 2018

nimroha force-pushed the nimroha:npz_file_IO branch from 58ecee6 to 770e433 May 17, 2018

This comment has been minimized.

nimroha commented May 17, 2018

hameerabbasi requested changes May 18, 2018 View changes

This comment has been minimized.

This comment has been minimized.

hameerabbasi May 18, 2018 • edited

This comment has been minimized.

This comment has been minimized.

hameerabbasi May 18, 2018 • edited

This comment has been minimized.

hameerabbasi May 18, 2018 • edited

This comment has been minimized.

hameerabbasi reviewed May 18, 2018 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hameerabbasi approved these changes May 20, 2018 View changes

Hide details View details hameerabbasi merged commit dfaee0c into pydata:master May 20, 2018 4 checks passed

4 checks passed

This comment has been minimized.

hameerabbasi commented May 20, 2018

This comment has been minimized.

nimroha commented May 20, 2018

nimroha commented May 17, 2018 •

edited by hameerabbasi

codecov-io commented May 17, 2018 •

edited

hameerabbasi requested changes May 17, 2018

View changes

nimroha force-pushed the nimroha:npz_file_IO branch from `58ecee6` to `770e433` May 17, 2018

hameerabbasi requested changes May 18, 2018

View changes

hameerabbasi May 18, 2018 •

edited

hameerabbasi May 18, 2018 •

edited

hameerabbasi May 18, 2018 •

edited

hameerabbasi reviewed May 18, 2018

View changes

hameerabbasi approved these changes May 20, 2018

View changes

hameerabbasi merged commit `dfaee0c` into pydata:master May 20, 2018
4 checks passed