[SYSTEMML-1116] Make SystemML Python DSL NumPy-friendly #290

niketanpansare · 2016-11-18T01:24:37Z

Bumped up the Python package version number @deroneriksson
Allow SystemML matrix to be invoked by numpy methods ( @dusenberrymw ):

import systemml as sml
import numpy as np
m1 = sml.matrix(np.ones((3,3)) + 2)
m2 = sml.matrix(np.ones((3,3)) + 3)
np.add(m1, m2).toNumPyArray()

Allow users to control the depth of lazy evaluation (as per @frreiss suggestion).

>>> import systemml as sml
>>> import numpy as np
>>> m1 = sml.matrix(np.ones((3,3)) + 2)

Welcome to Apache SystemML!

>>> m2 = sml.matrix(np.ones((3,3)) + 3)
>>> np.add(m1, m2) + m1
# This matrix (mVar4) is backed by below given PyDML script (which is not yet evaluated). To fetch the data of this matrix, invoke toNumPyArray() or toDataFrame() or toPandas() methods.
mVar2 = load(" ", format="csv")
mVar1 = load(" ", format="csv")
mVar3 = mVar1 + mVar2
mVar4 = mVar3 + mVar1
save(mVar4, " ")


>>> sml.set_max_depth(1)
>>> m1 = sml.matrix(np.ones((3,3)) + 2)
>>> m2 = sml.matrix(np.ones((3,3)) + 3)
>>> np.add(m1, m2) + m1
# This matrix (mVar8) is backed by NumPy array. To fetch the NumPy array, invoke toNumPyArray() method.

Few caveats:

The current version of NumPy explicitly disables overriding ufunc, but this should be enabled in next release. Until then to test above code, please use:

git clone https://github.com/niketanpansare/numpy.git
cd numpy
python setup.py install

Following NumPy ufunc that are not yet implemented in SystemML's Python DSL:

conj, hyperbolic/inverse-hyperbolic functions(i.e. sinh, arcsinh, cosh, ...), bitwise operator, xor operator, isreal, iscomplex, isfinite, isinf, isnan, copysign, nextafter, modf, frexp, trunc

dusenberrymw · 2016-11-18T23:10:53Z

This is quite interesting! Integrating the DSL more tightly with NumPy in this manner is great. I haven't tried it yet, but +1 to the example I see.

As for the depth of evaluation, what it we changed it to something simpler such as sml.set_lazy(False) to disable lazy evaluation?

niketanpansare · 2016-11-18T23:32:41Z

Thanks @dusenberrymw ... My intent is that if we can even scale subset of external libraries that uses numpy/scipy with minimal change (for eg: scikit-learn), it will be a great selling point for SystemML. Not sure if you noticed, Python SystemML package is much closer to a traditional python package, i.e. no jar needs to be specified in --driver-class-path of pyspark :)

sml.set_lazy(False) is much cleaner interface. I can update it in my next commit 👍

niketanpansare · 2016-11-19T21:24:07Z

Completed tasks in this PR:

Added python test cases for matrix (as per @asurve suggestion).
Added web documentation for all the Python APIs: http://niketanpansare.github.io/incubator-systemml/python-reference (as per @asurve suggestion).
Much tighter integration with NumPy (and several additional operators supported) via https://docs.scipy.org/doc/numpy-dev/reference/ufuncs.html
Facility to enable and disable lazy evaluation via set_lazy method (as per @dusenberrymw suggestion).
matrix class itself has almost all basic linear algebra operators supported by DML.
Update SystemML.jar to *-incubating.jar (as per @lresende suggestion).
Added maven cleanup logic (as per @deroneriksson suggestion).
Integrate python testcases with maven and jenkins: @dusenberrymw @akchinSTC, @asurve To run test on Jenkins:

Set the RUN_PYTHON_TEST flag in the org.apache.sysml.test.integration.functions.python.PythonTestRunner class to true. This is currently set to false until we ensure below two steps.
We will have to setup Spark and SPARK_HOME environment.
Make sure SystemML.jar is created in the target/ folder before tests are run (This should happen automatically).

Remaining tasks:

Large-scale testing: The current matrix has a huge performance issue (due to py4j conversion) especially in non-lazy setting or when evaluated too often. This issue was partly in this PR by reducing the explicit conversion during eval.
Pushdown-able loop structures.
Explore integration of non-unfuncs in Numpy (such as concatenate)
Refactor scikit-learn to make it scalable (by above point 5 and also by not redundantly forcing numpy array conversion).

@bertholdreinwald @dusenberrymw @deroneriksson @lresende @asurve If it's OK with you all, I can merge this in.

niketanpansare · 2016-11-20T07:49:28Z

Also, updated the issue https://issues.apache.org/jira/browse/SYSTEMML-1043

- Allow SystemML matrix to be invoked by numpy methods: import systemml as sml import numpy as np m1 = sml.matrix(np.ones((3,3)) + 2) m2 = sml.matrix(np.ones((3,3)) + 3) np.add(m1, m2).toNumPyArray() - Allow users to control the depth of lazy evaluation

python libraries

eliminating unnecessary java to python conversion if possible

dusenberrymw

@niketanpansare Okay I tried this out locally, and overall it's great! 👍 I'm really excited for this deeper integration with Python and NumPy specifically. Also +1 for the ability to pip install and then start up Spark normally without the --driver_class_path & --jars stuff.

A few thoughts:

Can you test with Python 3 as well (there are some print statements, for example, that have Python 2 syntax)?
Can you implement np.dot(...)? sml.matrix.dot is implemented, but the ufunc is missing for np.dot(m1,m2), where m1 and m2 are the systemml.defmatrix matrices: TypeError: __numpy_ufunc__ not implemented for this type.

Other things:

Can we update toNumPyArray() to toNumPy() for simplicity?
Can we update toDataFrame() to toDF() for simplicity and to be the same as Spark?

dusenberrymw · 2016-11-29T21:09:29Z

src/main/python/systemml/classloader.py

+        java_dir = os.path.join(imp.find_module("systemml")[1], "systemml-java")
+        for file in os.listdir(java_dir):
+            if fnmatch.fnmatch(file, 'systemml-*-incubating-SNAPSHOT.jar'):
+                jar_file_name = os.path.join(java_dir, file)


Have we tested if this JAR will be distributed to the worker nodes on a cluster as well?

Yes. Please try following script on your spark cluster:

import systemml as sml from systemml.random import uniform m1 = uniform(size=(10000,10000)) m2 = m1.dot(m1).sum().toNumPy() print(m2)

dusenberrymw · 2016-11-29T21:10:31Z

src/main/python/tests/test_matrix_agg_fn.py

+
+# To run:
+#   - Python 2: `PYSPARK_PYTHON=python2 spark-submit --master local[*] --driver-class-path SystemML.jar test_mlcontext.py`
+#   - Python 3: `PYSPARK_PYTHON=python3 spark-submit --master local[*] --driver-class-path SystemML.jar test_mlcontext.py`


Can you update these two test invocation lines with this filename?

dusenberrymw · 2016-11-29T21:11:10Z

src/main/python/tests/test_matrix_binary_op.py

+
+# To run:
+#   - Python 2: `PYSPARK_PYTHON=python2 spark-submit --master local[*] --driver-class-path SystemML.jar test_mlcontext.py`
+#   - Python 3: `PYSPARK_PYTHON=python3 spark-submit --master local[*] --driver-class-path SystemML.jar test_mlcontext.py`


Can you update these two test invocation lines with this filename?

dusenberrymw · 2016-11-29T21:11:24Z

src/main/python/tests/test_mllearn_df.py

+
+# To run:
+#   - Python 2: `PYSPARK_PYTHON=python2 spark-submit --master local[*] --driver-class-path SystemML.jar test_mllearn.py`
+#   - Python 3: `PYSPARK_PYTHON=python3 spark-submit --master local[*] --driver-class-path SystemML.jar test_mllearn.py`


Can you update these two test invocation lines with this filename?

dusenberrymw · 2016-11-29T21:13:31Z

src/test/java/org/apache/sysml/test/integration/functions/python/PythonTestRunner.java

+
+/**
+ * To run Python tests, please:
+ * 1. Set the RUN_PYTHON_TEST flag to true.


Let's aim to run this test by default on Jenkins.

dusenberrymw · 2016-11-29T21:17:23Z

src/main/python/systemml/defmatrix.py

-    # Don't use this method instead use matrix's printAST()
-    def printAST(self, numSpaces):
+    # Don't use this method instead use matrix's print_ast()
+    def print_ast(self, numSpaces):


Let's rename this to _print_ast to indicate that it is a private method not for use by the user.

dusenberrymw · 2016-11-29T21:23:41Z

src/main/python/systemml/defmatrix.py

+    """
+    Creates a single column vector with values starting from <start>, to <stop>, in increments of <step>.
+    Note: Unlike Numpy's arange which returns a row-vector, this returns a column vector.
+    Also, Unlike Numpy's arange which doesnot include stop, this method includes stop in the interval.


I would prefer to only support the equivalent of np.arange in this Python DSL, rather than introduce our R-like seq function. The users of the Python DSL will be Python users, so they will expect every aspect of our Python DSL to be Pythonic.

dusenberrymw · 2016-11-29T21:28:47Z

src/main/python/systemml/defmatrix.py

+
+    def __numpy_ufunc__(self, func, method, pos, inputs, **kwargs):
+        """
+        This function enables systemml matrix to be compatible with NumPy's ufuncs.


Can you add a bit of documentation to the parameters here?

dusenberrymw · 2016-11-29T21:35:20Z

src/main/python/tests/test_matrix_agg_fn.py

+        self.assertTrue(np.allclose(sml.matrix(m1).sum(axis=0), m1.sum(axis=0)))
+
+    def test_sum3(self):
+        self.assertTrue(np.allclose(sml.matrix(m1).sum(axis=1), m1.sum(axis=1).reshape(dim, 1)))


Can we add a quick note to the reference guide that explains that we always return a 2d matrix (ex. (3,1)), while NumPy can return a 1d vector (i.e. (3,))?

Also, ditto for the other functions below with this behavior.

dusenberrymw · 2016-11-29T21:41:01Z

src/main/python/tests/test_mllearn_df.py

+# ~/spark-1.6.1-scala-2.11/bin/spark-submit --master local[*] --driver-class-path SystemML.jar test.py
+class TestMLLearn(unittest.TestCase):
+
+    def testLogisticSK2(self):


Nitpick: To be in line with the Pythonic style of the rest of the DSL codebase, can we update these functions from CamelCase to underscore_case?

niketanpansare · 2016-11-29T22:57:26Z

Thanks @dusenberrymw for your review. I will incorporate your suggestion in the next commit :)

simplicity and to be the same as Spark

niketanpansare · 2016-12-02T18:23:04Z

Can you implement np.dot(...)?
Can we update toNumPyArray() to toNumPy() for simplicity?
Can we update toDataFrame() to toDF() for simplicity and to be the same as Spark?

Addressed in the commit afac2b2

As an FYI, I am addressing similar tasks in seperate commit to simplify your review process :)

niketanpansare · 2016-12-02T19:29:10Z

Addressed remaining comments in the commit: b4b8394

@dusenberrymw can you please review ? ... Let's try to push this PR in today if possible as #305 is dependent on this PR.

dusenberrymw

Okay just a few comments regarding print_ast and errors in Python 3 due to Python 2 print statements.

dusenberrymw · 2016-12-02T23:11:11Z

src/main/python/systemml/defmatrix.py

@@ -413,7 +417,7 @@ class matrix(object):
       Then the left-indexed matrix is set to be backed by DMLOp consisting of following pydml:
       left-indexed-matrix = new-deep-copied-matrix
       left-indexed-matrix[index] = value
-    8. Please use m.print_ast() and/or  type `m` for debugging. Here is a sample session:
+    8. Please use m._print_ast() and/or  type `m` for debugging. Here is a sample session:


The function name change for this internal function looks good, but I think the documentation is wrong now -- the user should still use the matrix print_ast() function, right?

dusenberrymw · 2016-12-02T23:21:28Z

src/main/python/systemml/defmatrix.py

-    # Don't use this method instead use matrix's printAST()
-    def printAST(self, numSpaces):
+    # Don't use this method instead use matrix's _print_ast()
+    def _print_ast(self, numSpaces):


Okay the name change for this function def _print_ast(self, numSpaces) is good since it shouldn't be called by the user. However, the other function that is supposed to be used instead should still be called print_ast() without an underscore. Otherwise, they both seem like internal functions.

dusenberrymw · 2016-12-02T23:25:51Z

src/main/python/systemml/defmatrix.py

        return self

-    def printAST(self, numSpaces = 0):
+    def _print_ast(self, numSpaces = 0):


Yeah let's make this function named print_ast() since it is intended to be called by the user, and the other variant that is not supposed to be called by the user can still be named _print_ast(numSpaces).

dusenberrymw · 2016-12-02T23:28:55Z

src/main/python/systemml/defmatrix.py

+            if matrix.THROW_ARRAY_CONVERSION_ERROR:
+                raise Exception('[ERROR]:' + msg)
+            else:
+                print '[WARN]:' + msg


Can you update this to Python 3 compatible syntax, as this currently throws an error as soon as the systemml package is imported.

dusenberrymw · 2016-12-02T23:29:01Z

src/main/python/systemml/defmatrix.py

-        out = matrix(None, op=dmlOp)
-        dmlOp.dml = [out.ID, ' = ', self.ID ] + getIndexingDML(index) + [ '\n' ]
-        return out
+        print '__getitem__'


Can you update this to Python 3 compatible syntax, as this currently throws an error as soon as the systemml package is imported.

Python 3 convention

niketanpansare · 2016-12-03T00:02:45Z

@dusenberrymw addressed above comments in the commit ecc242b

dusenberrymw

LGTM. This is an awesome update!

dusenberrymw · 2016-12-03T00:15:20Z

Follow up items:

Automatic Python 2 & 3 testing.
np.arange vs. DML seq (different semantics, Python people will prefer the former).

Also, we should closely follow the NumPy ufunc stuff. Looks like they are revisiting the idea, but may be renaming the function to array_ufunc [1]. The ideal case would be to not have to rely on a custom version of NumPy.

[1]: numpy/numpy#8247

niketanpansare · 2016-12-03T00:30:24Z

Once numpy/numpy#8247 is merged and numpy is released, we can update our setup to have that version as dependency :)

deroneriksson · 2016-12-03T00:35:34Z

Thank you @niketanpansare for all your hard work on this PR!

1. Added python test cases for matrix. 2. Added web documentation for all the Python APIs. 3. Added set_lazy method to enable and disable lazy evaluation. 4. matrix class itself has almost all basic linear algebra operators supported by DML. 4. Updated SystemML.jar to *-incubating.jar 5. Added maven cleanup logic for python artifacts. 6. Integrated python testcases with maven (See org.apache.sysml.test.integration.functions.python.PythonTestRunner). This requires SPARK_HOME to be set. Closes #290.

1. Added python test cases for matrix. 2. Added web documentation for all the Python APIs. 3. Added set_lazy method to enable and disable lazy evaluation. 4. matrix class itself has almost all basic linear algebra operators supported by DML. 4. Updated SystemML.jar to *-incubating.jar 5. Added maven cleanup logic for python artifacts. 6. Integrated python testcases with maven (See org.apache.sysml.test.integration.functions.python.PythonTestRunner). This requires SPARK_HOME to be set. Closes apache#290.

niketanpansare mentioned this pull request Nov 18, 2016

[SYSTEMML-1099] Mavenize the creation of python package #285

Closed

Niketan Pansare added 15 commits November 20, 2016 10:49

Updating the depth logic

01bc969

Adding array interface to allow systemml matrix to fit into existing

ae22c7b

python libraries

Supported shape functionality in systemml matrix

5de9d27

Updated array documentation

f636a92

Adding python documentation

97c22b9

Updating python doc

94b3532

Updating the documentation

e6cfadc

Updated the documentation

2a065f6

Added initial set of testcases for matrix

208e558

Updated the documentation

7979c93

Adding reference guide to index

b7dde35

Added load method and improved the performance of matrix class by

e3e5a2d

eliminating unnecessary java to python conversion if possible

Changing SystemML.jar to systemml-*-incubating-SNAPSHOT.jar for pip

f4317ec

Updating the documentation

29b3d7c

niketanpansare force-pushed the python_dsl branch from 136e436 to 29b3d7c Compare November 20, 2016 18:50

Niketan Pansare added 8 commits November 20, 2016 15:36

Added clean phase for python

8221c60

Updating the clean logic

d6d2add

Updating the clean logic

5a8e279

Added save function to matrix class

42cb130

Fixing random numpy failures

3e88ba7

Added python test

65bc040

Added kurtosis, skew, hstack, vstack, full,seq

41feda3

Removed kurtosis and updated the test cases for moment and variance

8afae22

dusenberrymw reviewed Nov 29, 2016

View reviewed changes

niketanpansare mentioned this pull request Dec 2, 2016

[SYSTEMML-1139] Tighten maven integration for python artifact #305

Closed

Updating toNumPyArray() to toNumPy() and toDataFrame() to toDF() for

afac2b2

simplicity and to be the same as Spark

Addressing Mike's comments

b4b8394

dusenberrymw reviewed Dec 2, 2016

View reviewed changes

Changed _print_ast according Python convention and print according

ecc242b

Python 3 convention

dusenberrymw approved these changes Dec 3, 2016

View reviewed changes

asfgit closed this in 23ccab8 Dec 3, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SYSTEMML-1116] Make SystemML Python DSL NumPy-friendly #290

[SYSTEMML-1116] Make SystemML Python DSL NumPy-friendly #290

niketanpansare commented Nov 18, 2016 •

edited

dusenberrymw commented Nov 18, 2016

niketanpansare commented Nov 18, 2016 •

edited

niketanpansare commented Nov 19, 2016 •

edited

niketanpansare commented Nov 20, 2016

dusenberrymw left a comment

dusenberrymw Nov 29, 2016

niketanpansare Dec 2, 2016

dusenberrymw Nov 29, 2016

dusenberrymw Nov 29, 2016

dusenberrymw Nov 29, 2016

dusenberrymw Nov 29, 2016

dusenberrymw Nov 29, 2016

dusenberrymw Nov 29, 2016

dusenberrymw Nov 29, 2016

dusenberrymw Nov 29, 2016

dusenberrymw Nov 29, 2016

dusenberrymw Nov 29, 2016

niketanpansare commented Nov 29, 2016

niketanpansare commented Dec 2, 2016 •

edited

niketanpansare commented Dec 2, 2016

dusenberrymw left a comment

dusenberrymw Dec 2, 2016

dusenberrymw Dec 2, 2016

dusenberrymw Dec 2, 2016

dusenberrymw Dec 2, 2016

dusenberrymw Dec 2, 2016

niketanpansare commented Dec 3, 2016

dusenberrymw left a comment •

edited

dusenberrymw commented Dec 3, 2016 •

edited

niketanpansare commented Dec 3, 2016

deroneriksson commented Dec 3, 2016

[SYSTEMML-1116] Make SystemML Python DSL NumPy-friendly #290

[SYSTEMML-1116] Make SystemML Python DSL NumPy-friendly #290

Conversation

niketanpansare commented Nov 18, 2016 • edited

dusenberrymw commented Nov 18, 2016

niketanpansare commented Nov 18, 2016 • edited

niketanpansare commented Nov 19, 2016 • edited

niketanpansare commented Nov 20, 2016

dusenberrymw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niketanpansare commented Nov 29, 2016

niketanpansare commented Dec 2, 2016 • edited

niketanpansare commented Dec 2, 2016

dusenberrymw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niketanpansare commented Dec 3, 2016

dusenberrymw left a comment • edited

Choose a reason for hiding this comment

dusenberrymw commented Dec 3, 2016 • edited

niketanpansare commented Dec 3, 2016

deroneriksson commented Dec 3, 2016

niketanpansare commented Nov 18, 2016 •

edited

niketanpansare commented Nov 18, 2016 •

edited

niketanpansare commented Nov 19, 2016 •

edited

niketanpansare commented Dec 2, 2016 •

edited

dusenberrymw left a comment •

edited

dusenberrymw commented Dec 3, 2016 •

edited