Add tutorial Gotchas using NumPy #12007

Ishitori · 2018-08-02T19:41:05Z

Description

Add tutorial Gotchas using NumPy. Explain about blocking calls and how to minimize the impact.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Code is well-documented:
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Adds new tutorial

larroy · 2018-08-02T21:34:29Z

docs/tutorials/gluon/gotchas_numpy_in_mxnet.md

+
+One key difference is in the way calculations are executed. Every `NDArray` manipulation in Apache MXNet is done in asynchronous, non-blocking way. That means, that when we write code like `c = a * b`, where both `a` and `b` are `NDArrays`, the function got pushed to the [Execution Engine](https://mxnet.incubator.apache.org/architecture/overview.html#execution-engine), which starts the calculation. The function immediately returns back, and the  user thread can continue execution, despite the fact that the calculation may not have been completed yet. 
+
+`Execution Engine` builds computation graph which may reorder or combine some calculations, but it honors dependency order: if there are other manipulation with `c` done later in the code, the `Execution Engine` will start doing them once the result of `c` is available. We don't need to write callbacks to start execution of subsequent code - the `Execution Engine` is going to do it for us. 


builds the

rahul003 · 2018-08-03T06:43:30Z

docs/tutorials/gluon/gotchas_numpy_in_mxnet.md

+
+## NumPy operators vs. NDArray operators
+
+Despite the fact that [NDArray API](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html) was specifically designed to be similar to `NumPy`, sometimes it is not easy to replace existing `NumPy` computations. The main reason is that not all operators, that are available in `NumPy`, are available in `NDArray API`. [This regularly updated page](https://github.com/apache/incubator-mxnet/issues/3199) contains the list of `NDArray API`  operators in progress, where:


The page linked doesn't seem to be updated in 2 years. Is that the only reference we can provide? We should probably at least remove the words 'regularly updated'

I don't think you need to mention the Issue. You can redirect to the API page
http://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#the-ndarray-class

Why isn't that info moved over to Confluence?

nswamy

Thanks for this really useful document. Looks good,
tagging @aaronmarkham for reviewing readability

nswamy · 2018-08-08T23:59:45Z

docs/tutorials/gluon/gotchas_numpy_in_mxnet.md

+
+## NumPy operators vs. NDArray operators
+
+Despite the fact that [NDArray API](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html) was specifically designed to be similar to `NumPy`, sometimes it is not easy to replace existing `NumPy` computations. The main reason is that not all operators, that are available in `NumPy`, are available in `NDArray API`. [This regularly updated page](https://github.com/apache/incubator-mxnet/issues/3199) contains the list of `NDArray API`  operators in progress, where:


I don't think you need to mention the Issue. You can redirect to the API page
http://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html#the-ndarray-class

sandeep-krishnamurthy · 2018-08-15T16:32:17Z

@aaronmarkham - ping

aaronmarkham

Nice tutorial! Just a few suggestions.

aaronmarkham · 2018-08-09T15:36:11Z

docs/tutorials/gluon/gotchas_numpy_in_mxnet.md

+
+Instead of using NumPy arrays Apache MXNet offers its own array implementation named [NDArray](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html). `NDArray API` was intentionally designed to be similar to `NumPy`, but there are differences.
+
+One key difference is in the way calculations are executed. Every `NDArray` manipulation in Apache MXNet is done in asynchronous, non-blocking way. That means, that when we write code like `c = a * b`, where both `a` and `b` are `NDArrays`, the function got pushed to the [Execution Engine](https://mxnet.incubator.apache.org/architecture/overview.html#execution-engine), which starts the calculation. The function immediately returns back, and the  user thread can continue execution, despite the fact that the calculation may not have been completed yet. 


nit: got --> was or is

aaronmarkham · 2018-08-15T19:50:34Z

docs/tutorials/gluon/gotchas_numpy_in_mxnet.md

+
+To get the result of the computation we only need to access the resulting variable, and the flow of the code will be blocked until the computation results are assigned to the resulting variable. This behavior allows to increase code performance while still supporting imperative programming mode. 
+
+Refer to [this tutorial](https://mxnet.incubator.apache.org/tutorials/basic/ndarray.html), if you are new to Apache MXNet and would like to learn more how to manipulate NDArrays.


I prefer to link with specific information to boost the performance of indexing and SEO. So... Refer to the intro tutorial to NDArray if you are new to MXNet...

aaronmarkham · 2018-08-15T19:55:00Z

docs/tutorials/gluon/gotchas_numpy_in_mxnet.md

+
+## NumPy operators vs. NDArray operators
+
+Despite the fact that [NDArray API](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html) was specifically designed to be similar to `NumPy`, sometimes it is not easy to replace existing `NumPy` computations. The main reason is that not all operators, that are available in `NumPy`, are available in `NDArray API`. [This regularly updated page](https://github.com/apache/incubator-mxnet/issues/3199) contains the list of `NDArray API`  operators in progress, where:


Why isn't that info moved over to Confluence?

aaronmarkham · 2018-08-15T19:58:22Z

docs/tutorials/gluon/gotchas_numpy_in_mxnet.md

+    <NDArray 10 @cpu(0)> <!--notebook-skip-line-->
+
+
+### Search for an operator on [Github](https://github.com/apache/incubator-mxnet/pulls)


What about using the label?
https://github.com/apache/incubator-mxnet/labels/Operator
Or add closed to it?

sandeep-krishnamurthy · 2018-08-20T20:56:23Z

@lshitori - Can you please address few suggested changes by @aaronmarkham and this is all ready to go!

Ishitori · 2018-08-29T12:31:37Z

@sandeep-krishnamurthy @aaronmarkham @larroy @rahul003 thanks for the review. Updated the tutorial based on your comments

* Add tutorial Gotchas using NumPy * Forcing build * Code review fix * Forcing build

Add tutorial Gotchas using NumPy

a5f58d3

Ishitori requested a review from szha as a code owner August 2, 2018 19:41

Forcing build

c66fd75

larroy reviewed Aug 2, 2018

View reviewed changes

rahul003 reviewed Aug 3, 2018

View reviewed changes

nswamy reviewed Aug 9, 2018

View reviewed changes

nswamy added Feature Doc NDArray pr-awaiting-review PR is waiting for code review labels Aug 9, 2018

szha removed their request for review August 9, 2018 19:05

aaronmarkham approved these changes Aug 15, 2018

View reviewed changes

sandeep-krishnamurthy added pr-awaiting-response PR is reviewed and waiting for contributor to respond and removed pr-awaiting-review PR is waiting for code review labels Aug 20, 2018

Sergey Sokolov added 2 commits August 28, 2018 12:28

Code review fix

5be2f91

Forcing build

1fa8c29

sandeep-krishnamurthy merged commit 1f0d6ba into apache:master Aug 29, 2018

anirudh2290 pushed a commit to anirudh2290/mxnet that referenced this pull request Sep 19, 2018

Add tutorial Gotchas using NumPy (apache#12007)

ec6b32d

* Add tutorial Gotchas using NumPy * Forcing build * Code review fix * Forcing build

szha added Feature request and removed Feature labels Nov 14, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tutorial Gotchas using NumPy #12007

Add tutorial Gotchas using NumPy #12007

Ishitori commented Aug 2, 2018

larroy Aug 2, 2018

rahul003 Aug 3, 2018

nswamy Aug 8, 2018

aaronmarkham Aug 15, 2018

nswamy left a comment

nswamy Aug 8, 2018

sandeep-krishnamurthy commented Aug 15, 2018

aaronmarkham left a comment

aaronmarkham Aug 9, 2018

aaronmarkham Aug 15, 2018

aaronmarkham Aug 15, 2018

aaronmarkham Aug 15, 2018

sandeep-krishnamurthy commented Aug 20, 2018

Ishitori commented Aug 29, 2018


		One key difference is in the way calculations are executed. Every `NDArray` manipulation in Apache MXNet is done in asynchronous, non-blocking way. That means, that when we write code like `c = a * b`, where both `a` and `b` are `NDArrays`, the function got pushed to the [Execution Engine](https://mxnet.incubator.apache.org/architecture/overview.html#execution-engine), which starts the calculation. The function immediately returns back, and the user thread can continue execution, despite the fact that the calculation may not have been completed yet.

		`Execution Engine` builds computation graph which may reorder or combine some calculations, but it honors dependency order: if there are other manipulation with `c` done later in the code, the `Execution Engine` will start doing them once the result of `c` is available. We don't need to write callbacks to start execution of subsequent code - the `Execution Engine` is going to do it for us.


		## NumPy operators vs. NDArray operators

		Despite the fact that [NDArray API](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html) was specifically designed to be similar to `NumPy`, sometimes it is not easy to replace existing `NumPy` computations. The main reason is that not all operators, that are available in `NumPy`, are available in `NDArray API`. [This regularly updated page](https://github.com/apache/incubator-mxnet/issues/3199) contains the list of `NDArray API` operators in progress, where:


		Instead of using NumPy arrays Apache MXNet offers its own array implementation named [NDArray](https://mxnet.incubator.apache.org/api/python/ndarray/ndarray.html). `NDArray API` was intentionally designed to be similar to `NumPy`, but there are differences.

		One key difference is in the way calculations are executed. Every `NDArray` manipulation in Apache MXNet is done in asynchronous, non-blocking way. That means, that when we write code like `c = a * b`, where both `a` and `b` are `NDArrays`, the function got pushed to the [Execution Engine](https://mxnet.incubator.apache.org/architecture/overview.html#execution-engine), which starts the calculation. The function immediately returns back, and the user thread can continue execution, despite the fact that the calculation may not have been completed yet.


		To get the result of the computation we only need to access the resulting variable, and the flow of the code will be blocked until the computation results are assigned to the resulting variable. This behavior allows to increase code performance while still supporting imperative programming mode.

		Refer to [this tutorial](https://mxnet.incubator.apache.org/tutorials/basic/ndarray.html), if you are new to Apache MXNet and would like to learn more how to manipulate NDArrays.

		<NDArray 10 @cpu(0)> <!--notebook-skip-line-->


		### Search for an operator on [Github](https://github.com/apache/incubator-mxnet/pulls)

Add tutorial Gotchas using NumPy #12007

Add tutorial Gotchas using NumPy #12007

Conversation

Ishitori commented Aug 2, 2018

Description

Checklist

Essentials

Changes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nswamy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sandeep-krishnamurthy commented Aug 15, 2018

aaronmarkham left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sandeep-krishnamurthy commented Aug 20, 2018

Ishitori commented Aug 29, 2018