Adding support for saving matrices to files in npz format #154

nimroha · 2018-05-17T14:24:39Z

Closes #153

100% test coverage python 2.7 and 3.5

added all the docs but could not build HTML locally

codecov-io · 2018-05-17T14:26:12Z

Codecov Report

Merging #154 into master will increase coverage by 0.04%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #154      +/-   ##
==========================================
+ Coverage   96.89%   96.93%   +0.04%     
==========================================
  Files          10       11       +1     
  Lines        1191     1207      +16     
==========================================
+ Hits         1154     1170      +16     
  Misses         37       37

Impacted Files	Coverage Δ
sparse/io.py	`100% <100%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f939529...d5c19a7. Read the comment docs.

hameerabbasi

Overall, this is an awesome effort. We strive for perfection, so you can make changes and I'll approve. 😄

One thing not in this review is that you should add load_npz and save_npz into the list of functions in docs/generated/sparse.rst before building the docs.

hameerabbasi · 2018-05-17T14:34:06Z

sparse/io.py

+            [0.        , 0.86522495]]])
+    >>> os.remove('mat.npz')
+
+    :param filename: string or file


Parameters must be documented Numpy-style. See https://numpydoc.readthedocs.io/en/latest/format.html

If you're using PyCharm, see "Docstring format" https://www.jetbrains.com/help/pycharm/settings-tools-python-integrated-tools.html

hameerabbasi · 2018-05-17T14:34:21Z

sparse/io.py

+            [0.        , 0.86522495]]])
+    >>> os.remove('mat.npz')
+
+    :param filename: file-like object, string, or pathlib.Path


See comment above.

hameerabbasi · 2018-05-17T14:35:22Z

sparse/tests/test_io.py

+from sparse.utils import assert_eq
+
+
+def test_save_load_npz_file():


Try parametrizing compressed as True/False. See example parametrizations in test_coo.py.

hameerabbasi · 2018-05-17T14:38:09Z

sparse/tests/test_io.py

+    assert_eq(x, z)
+    assert_eq(y, z.todense())
+
+    # test exception on wrong format


Make this into another test.

hameerabbasi · 2018-05-17T14:39:02Z

sparse/io.py

+    """
+
+    nodes = {'data':   matrix.data,
+             'coords': matrix.coords}


You want to store the shape too, sometimes these aren't enough on their own.

hameerabbasi · 2018-05-17T14:42:13Z

sparse/io.py

@@ -0,0 +1,96 @@
+import numpy as np
+
+from sparse.coo.core import COO


This should be .coo.core. Inside a package, we should always use relative imports.

hameerabbasi · 2018-05-17T14:46:02Z

docs/generated/sparse.io.load_npz.rst

@@ -0,0 +1,6 @@
+io.load_npz


Remove this file and build the docs. It will be auto-generated, then you can do git add.

hameerabbasi · 2018-05-17T14:46:13Z

docs/generated/sparse.io.save_npz.rst

@@ -0,0 +1,6 @@
+io.save_npz


Same as above.

hameerabbasi · 2018-05-17T14:46:36Z

sparse/tests/test_io.py

+    with pytest.raises(RuntimeError):
+        load_npz(filename)
+
+    shutil.rmtree(dir_name)


Try os.rmdir

Oh, ignore. I just read the docs on that... It doesn't remove unless empty.

hameerabbasi · 2018-05-17T14:48:23Z

sparse/io.py

+        except KeyError:
+            raise RuntimeError('The file {} does not contain a valid sparse matrix'.format(filename))
+
+    return COO(coords=coords, data=data)


Add sorted=True, has_duplicates=False and shape. The first two will make it a lot faster, and shape ensures consistency.

hameerabbasi · 2018-05-17T18:00:59Z

If this is too much for you, let me know and I'll fix it up. 🙂 The PR will still be in your name.

nimroha · 2018-05-17T18:29:15Z

no worries, just got caught up at work.
will work on it tonight

nimroha · 2018-05-17T22:27:22Z

addressed all your comments and added the doc files

hameerabbasi

Looks like most of the major concerns have been addressed, here are some other recommendations. Good work by the way. 😄

hameerabbasi · 2018-05-18T04:07:17Z

sparse/io.py

+        Whether to save in compressed or uncompressed mode
+
+    Returns
+    -------


Remove this section, as the function isn't returning anything.

hameerabbasi · 2018-05-18T04:09:47Z

sparse/tests/test_io.py

+
+import sparse
+
+from sparse.io import save_npz, load_npz


This still needs to be addressed. In scipy.sparse, the functions are in scipy.sparse, not scipy.sparse.io.

Therefore, here, they should be in sparse directly, not sparse.io. You need to import them from sparse here, not sparse.io.

hameerabbasi · 2018-05-18T04:10:26Z

sparse/tests/test_io.py

+    with pytest.raises(RuntimeError):
+        load_npz(filename)
+
+    shutil.rmtree(dir_name)


flake8 is failing because of a missing newline here.

hameerabbasi · 2018-05-18T04:13:54Z

sparse/io.py

+           [[0.        , 0.        ],
+            [0.        , 0.86522495]]])
+    >>> os.remove('mat.npz')
+


Optional, but would be really nice to have: A see also section with load_npz, np.savez and scipy.sparse.save_npz. Also a note saying that it's binary incompatible. See examples in our code and here: https://numpydoc.readthedocs.io/en/latest/format.html

hameerabbasi · 2018-05-18T04:16:13Z

sparse/io.py

+           [[0.        , 0.        ],
+            [0.        , 0.86522495]]])
+    >>> os.remove('mat.npz')
+


Same as above:

Optional, but would be really nice to have: A see also section with save_npz, np.load and scipy.sparse.load_npz. Also a note saying that it's binary incompatible.

hameerabbasi · 2018-05-18T04:17:02Z

docs/generated/sparse.rst

@@ -53,3 +53,6 @@ API

    where

+    save_npz


Both of these should be added in alphabetical order, otherwise they're hard to find.

hameerabbasi

A few more changes.

We want the user-facing API to be compatible with scipy.sparse, so the .io should be removed everywhere and it should just be treated as an internal submodule.

hameerabbasi · 2018-05-18T04:44:03Z

sparse/io.py

+    >>> mat
+    <COO: shape=(2, 2, 2), dtype=float64, nnz=2>
+    >>> sparse.io.save_npz('mat.npz', mat)
+    >>> loaded_mat = sparse.io.load_npz('mat.npz')


Change this to sparse.load_npz

hameerabbasi · 2018-05-18T04:44:44Z

sparse/io.py

+    >>> mat = sparse.COO(dense_mat)
+    >>> mat
+    <COO: shape=(2, 2, 2), dtype=float64, nnz=2>
+    >>> sparse.io.save_npz('mat.npz', mat)


Change this to sparse.save_npz

hameerabbasi · 2018-05-18T04:46:35Z

sparse/io.py

+    >>> mat
+    <COO: shape=(2, 2, 2), dtype=float64, nnz=2>
+    >>> sparse.io.save_npz('mat.npz', mat)
+    >>> loaded_mat = sparse.io.load_npz('mat.npz')


Same comments as above. Also, it would be best to remove the examples section and in the main description, just add

See :obj:`save_npz` for usage examples.

hameerabbasi · 2018-05-20T13:45:45Z

This is in! Thanks, @nimroha!

nimroha · 2018-05-20T14:42:34Z

thank you! pleasure working with you

…

On Sun, May 20, 2018, 16:45 Hameer Abbasi ***@***.***> wrote: This is in! Thanks, @nimroha <https://github.com/nimroha>! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#154 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APXkV-ORzdvI59Af740eEnn13VHn_BeNks5t0XOKgaJpZM4UDK5E> .

hameerabbasi requested changes May 17, 2018

View reviewed changes

NimrodM and others added 7 commits May 18, 2018 01:13

Added save_npz and load_npz methods

e9c735d

Added unittest

28c2fed

flake8 fixes, made test python2.7 compatible

9a6c38e

Added docstrings

2c8d94a

exposed methods in __init__, cosmetics in test

c394c0b

Added docs to /docs/generated

7b8d9da

fixed PR comments

770e433

nimroha force-pushed the npz_file_IO branch from 58ecee6 to 770e433 Compare May 17, 2018 22:16

added .rst files

34aec2c

hameerabbasi requested changes May 18, 2018

View reviewed changes

hameerabbasi reviewed May 18, 2018

View reviewed changes

fixed more PR comments

d5c19a7

hameerabbasi approved these changes May 20, 2018

View reviewed changes

hameerabbasi merged commit dfaee0c into pydata:master May 20, 2018

Adding support for saving matrices to files in npz format #154

Adding support for saving matrices to files in npz format #154

nimroha commented May 17, 2018 •

edited by hameerabbasi

codecov-io commented May 17, 2018 •

edited

hameerabbasi left a comment

hameerabbasi May 17, 2018

hameerabbasi May 17, 2018

hameerabbasi May 17, 2018

hameerabbasi May 17, 2018

hameerabbasi May 17, 2018

hameerabbasi May 17, 2018

hameerabbasi May 17, 2018

hameerabbasi May 17, 2018

hameerabbasi May 17, 2018

hameerabbasi May 17, 2018

hameerabbasi May 17, 2018

hameerabbasi May 17, 2018

hameerabbasi commented May 17, 2018

nimroha commented May 17, 2018

nimroha commented May 17, 2018

hameerabbasi left a comment

hameerabbasi May 18, 2018

hameerabbasi May 18, 2018 •

edited

hameerabbasi May 18, 2018

hameerabbasi May 18, 2018 •

edited

hameerabbasi May 18, 2018 •

edited

hameerabbasi May 18, 2018

hameerabbasi left a comment

hameerabbasi May 18, 2018

hameerabbasi May 18, 2018

hameerabbasi May 18, 2018

hameerabbasi commented May 20, 2018

nimroha commented May 20, 2018 via email

		from sparse.utils import assert_eq


		def test_save_load_npz_file():

		@@ -0,0 +1,96 @@
		import numpy as np

		from sparse.coo.core import COO

Adding support for saving matrices to files in npz format #154

Adding support for saving matrices to files in npz format #154

Conversation

nimroha commented May 17, 2018 • edited by hameerabbasi

codecov-io commented May 17, 2018 • edited

Codecov Report

hameerabbasi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hameerabbasi commented May 17, 2018

nimroha commented May 17, 2018

nimroha commented May 17, 2018

hameerabbasi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hameerabbasi May 18, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hameerabbasi May 18, 2018 • edited

Choose a reason for hiding this comment

hameerabbasi May 18, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hameerabbasi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hameerabbasi commented May 20, 2018

nimroha commented May 20, 2018 via email

nimroha commented May 17, 2018 •

edited by hameerabbasi

codecov-io commented May 17, 2018 •

edited

hameerabbasi May 18, 2018 •

edited

hameerabbasi May 18, 2018 •

edited

hameerabbasi May 18, 2018 •

edited