[numpy] numpy einsum #15493

hzfan · 2019-07-09T08:05:58Z

Description

numpy compatible einsum

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

add einsum
add einsum doc
add path optimization for imperative mode
add benchmark
add forward and backward test

Comments

support both explicit and implicit mode
support multiple operands
support broadcast
support 0-dim NDArray and 0-size NDArray

TODO

translate einsum_path into c, so that optimization can work in symbol mode
take advantage of tensordot when it is merged
support "greedy" optimization
support "optimal" optimization

Thank @reminisce and @haojin2 for guidance and review.

hzfan · 2019-07-09T08:08:04Z

@mxnet-label-bot add [Numpy]

reminisce · 2019-07-10T00:10:43Z

python/mxnet/numpy/multiarray.py

+    Chained array operations. For more complicated contractions, speed ups
+    might be achieved by repeatedly computing a 'greedy' path or pre-computing the
+    'optimal' path and repeatedly applying it, using an
+    `einsum_path` insertion (since version 1.12.0). Performance improvements can be


1.12.0 is the official numpy version. We'd better not include that.

reminisce · 2019-07-10T00:11:40Z

python/mxnet/numpy/multiarray.py

+    `einsum_path` insertion (since version 1.12.0). Performance improvements can be
+    particularly significant with larger arrays:
+
+    >>> a = np.ones(64).reshape(2,4,8)


Is the benchmark from the mxnet.numpy.einsum or the official np.einsum?

I haven't benchmarked yet. The benchmark here is from official np.einsum. I will remove them.

reminisce · 2019-07-10T00:50:27Z

src/operator/numpy/np_einsum_op-inl.h

+  int newdim;
+
+  const TShape& shape = op.shape_;
+  TShape stride;


Combine this line and the next line into one line.

reminisce · 2019-07-10T00:51:56Z

src/operator/numpy/np_einsum_op-inl.h

+                                   TShape* newshape,
+                                   TShape* newstride) {
+  using namespace mxnet_op;
+  int idim, ndim, icombine, combineoffset;


Define the variables where they are first used, i.e. narrow down the scope of variables as much as possible.

reminisce · 2019-07-10T01:18:34Z

src/operator/numpy/np_einsum_op.cc

+namespace mxnet {
+namespace op {
+
+inline bool NumpyEinsumShape(const nnvm::NodeAttrs& attrs,


nit: no need to inline

haojin2 · 2019-07-10T21:17:12Z

src/operator/numpy/np_einsum_op-inl.h

+#define NPY_MAXDIMS 32
+#define NPY_MAXARGS 32
+
+


one less blank line here.

haojin2 · 2019-07-10T21:17:30Z

src/operator/numpy/np_einsum_op-inl.h

+  return ret;
+}
+
+


one less blank line here

haojin2 · 2019-07-10T21:40:30Z

src/operator/numpy/np_einsum_op.cc

+  //   std::cout << output_labels[i] << " ";
+  // }
+  // std::cout << std::endl;
+  // std::cout << "max_broadcast = " << max_broadcast << std::endl;


remove dead code

hzfan · 2019-07-18T07:42:59Z

cherry-picked from #15565. Tensordot optimization has been enabled. @reminisce

* Fix backward_clip num inputs and type of clip params * Clip test * Trigger CI * Changes to clip docs * Fix docstring * Trigger CI

* Modify ndarray slice to have numpy compatbile behaviou * Minor syntax fix * Fix slice inconsistency * Allow empty outputs after slicing ndarrays * Fix

* remove test images * add script and .gitignore * add test helper to download images * remove unlicensed pic * add license header

* Implements tensordot and dot. * Change tests. * Add spaces. * Reorganize codes. * Remove np_matrix_op.h

* Adding Large Index Support for slice operator * adding changes to fix py2 related error in CI/CD * fixing base.py * rearrange system call and slower Feature() call * refactoring c_api, c_symbolic_api, c_api_common * templatizing code * caching results of runtime features and minor refactoring * fixing local caching in ndarray shape

* Remove old cudnn support (<v7). * Simplify conv impl selection. * Remove comments justifying STATIC_ASSERT_CUDNN_VERSION_GE.

* Add squeeze/flatten/transpose/reshape Add more ops back Add new files Add unit tests Add unit test for broadcast_arrays Add more ops and tests Add arange * clean up

* test rdiv * floating_point exception handle * add 10 other ops * added rpow and made numpy consistent * attempt to solve memory issue * linting fix * Trigger notification * lint

) * Allow operators with multiple outputs in get_atomic_symbol * Added unittest

* random ops * replace array with uniform * remove dtype * randn add * add multinomial * multi,randn small fix * add negative bino * fix memory issue - Failed to allocate CPU Memory * Trigger notification * linting fix * Trigger notification

* sequence_last, sequence_reverse, sequence_mask * working softmax_cross_entropy * fix linting, add index_copy * add softmax output * add leaky relu * add pooling * add layernorm * add dropout, activation, batchnorm and update layernorm * address comments to remove some comments * handling imports

* numpy-compatible concatenate upstream * extend ci deadline

* Fix ConcatType and add test * Remove return false * Change error message * Run RNN test only when CUDNN enabled * set default context for test_contrib_amp

hzfan · 2019-08-15T14:53:51Z

PRed to the master branch. Closing this one.

hzfan requested a review from szha as a code owner July 9, 2019 08:05

marcoabreu added the Numpy label Jul 9, 2019

hzfan force-pushed the einsum_pr branch from 31e8833 to 58846a7 Compare July 9, 2019 12:39

reminisce reviewed Jul 10, 2019

View reviewed changes

haojin2 reviewed Jul 10, 2019

View reviewed changes

hzfan force-pushed the einsum_pr branch 2 times, most recently from 4d2114e to cdfaf1c Compare July 16, 2019 07:18

reminisce force-pushed the numpy branch from d8d6b3b to 7da1a11 Compare July 17, 2019 08:51

haojin2 force-pushed the numpy branch from 7da1a11 to 76afc2d Compare July 18, 2019 00:00

reminisce mentioned this pull request Jul 18, 2019

Numpy Fix tensordot bug #15565

Closed

5 tasks

hzfan force-pushed the einsum_pr branch 2 times, most recently from 924d327 to b983ab4 Compare July 18, 2019 05:31

hzfan force-pushed the einsum_pr branch from c43ff9b to 05a233c Compare July 18, 2019 08:22

haojin2 force-pushed the numpy branch 2 times, most recently from 10a03fa to a23120e Compare July 23, 2019 01:11

hzfan force-pushed the einsum_pr branch 2 times, most recently from 7bcdcb3 to be06bd6 Compare July 24, 2019 17:49

haojin2 force-pushed the numpy branch from a23120e to 07d7a94 Compare July 24, 2019 18:59

hzfan force-pushed the einsum_pr branch from be06bd6 to 01c0012 Compare July 24, 2019 23:51

hzfan requested review from anirudh2290 and eric-haibin-lin as code owners July 24, 2019 23:51

hzfan force-pushed the einsum_pr branch 4 times, most recently from c301605 to 0114f6f Compare July 26, 2019 11:51

haojin2 force-pushed the numpy branch from 07d7a94 to 47f4cd3 Compare July 26, 2019 18:35

ptrendx and others added 25 commits August 9, 2019 16:05

Fix backward_clip num inputs and type of clip params (apache#15688)

0eb213d

* Fix backward_clip num inputs and type of clip params * Clip test * Trigger CI * Changes to clip docs * Fix docstring * Trigger CI

[Numpy] Numpy compatible slicing (apache#15798)

44a7fca

* Modify ndarray slice to have numpy compatbile behaviou * Minor syntax fix * Fix slice inconsistency * Allow empty outputs after slicing ndarrays * Fix

Making Features as a singleton for improved caching (apache#15835)

614cba3

Clojure package remove source images (apache#15828)

57927a9

* remove test images * add script and .gitignore * add test helper to download images * remove unlicensed pic * add license header

Numpy Tensordot and Dot Operator (apache#15820)

c3f5eea

* Implements tensordot and dot. * Change tests. * Add spaces. * Reorganize codes. * Remove np_matrix_op.h

fixing problem with existing Singleton Caching (apache#15868)

bd32de4

numpy linspace (apache#15852)

795990b

tvm infra for op attrs (apache#15854)

67daae7

cuDNN support cleanup (apache#15812)

c81535c

* Remove old cudnn support (<v7). * Simplify conv impl selection. * Remove comments justifying STATIC_ASSERT_CUDNN_VERSION_GE.

Port several np ops to master (apache#15867)

39bf4e0

* Add squeeze/flatten/transpose/reshape Add more ops back Add new files Add unit tests Add unit test for broadcast_arrays Add more ops and tests Add arange * clean up

Add large tensor support binary arithmetic (apache#15785)

11ce2a2

* test rdiv * floating_point exception handle * add 10 other ops * added rpow and made numpy consistent * attempt to solve memory issue * linting fix * Trigger notification * lint

Allow operators with multiple outputs in get_atomic_symbol (apache#15740

24a5cf0

) * Allow operators with multiple outputs in get_atomic_symbol * Added unittest

numpy-compatible split upstream (apache#15841)

f32b58e

Numpy-compatible concatenate upstream (apache#15894)

3d8035b

* numpy-compatible concatenate upstream * extend ci deadline

Numpy-compatible concatenate upstream (apache#15894)

78733f0

* numpy-compatible concatenate upstream * extend ci deadline

Numpy-compatible concatenate upstream (apache#15894)

b98f297

* numpy-compatible concatenate upstream * extend ci deadline

Numpy-compatible concatenate upstream (apache#15894)

9dbfc2d

* numpy-compatible concatenate upstream * extend ci deadline

Fix ConcatType backward type inference (apache#15829)

40593c6

* Fix ConcatType and add test * Remove return false * Change error message * Run RNN test only when CUDNN enabled * set default context for test_contrib_amp

einsum with optimization for imperative

cd25d47

fix tensordot temp space overwrite error

2d713d2

enable symbolic forward optimization with einsum_path

b8a70c2

enable symbolic backward optimization with einsum_path

3255b3e

hzfan force-pushed the einsum_pr branch from 38e27fc to 3255b3e Compare August 15, 2019 14:33

hzfan requested review from gigasquid and nswamy as code owners August 15, 2019 14:33

hzfan closed this Aug 15, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[numpy] numpy einsum #15493

[numpy] numpy einsum #15493

hzfan commented Jul 9, 2019 •

edited

hzfan commented Jul 9, 2019

reminisce Jul 10, 2019

hzfan Jul 16, 2019

reminisce Jul 10, 2019

hzfan Jul 16, 2019

reminisce Jul 10, 2019

hzfan Jul 16, 2019

reminisce Jul 10, 2019

hzfan Jul 16, 2019

reminisce Jul 10, 2019

hzfan Jul 16, 2019

haojin2 Jul 10, 2019

hzfan Jul 16, 2019

haojin2 Jul 10, 2019

hzfan Jul 16, 2019

haojin2 Jul 10, 2019

hzfan Jul 16, 2019

hzfan commented Jul 18, 2019

hzfan commented Aug 15, 2019

		#define NPY_MAXDIMS 32
		#define NPY_MAXARGS 32

[numpy] numpy einsum #15493

[numpy] numpy einsum #15493

Conversation

hzfan commented Jul 9, 2019 • edited

Description

Checklist

Essentials

Changes

Comments

TODO

hzfan commented Jul 9, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hzfan commented Jul 18, 2019

hzfan commented Aug 15, 2019

hzfan commented Jul 9, 2019 •

edited