[MXNET-432] Add Foreach #11531

zheng-da · 2018-07-02T18:44:30Z

Description

This PR is a copy of #10451.

This PR reconstructs MXNet IR system to support control flow operators and adds one control flow operator: foreach. It takes a Python function as input and run the function over the elements in the input array. foreach is similar to scan in TensorFlow.

This PR is part of the proposal of adding a set of control flow operators to MXNet.
https://cwiki.apache.org/confluence/display/MXNET/Optimize+dynamic+neural+network+models+with+control+flow+operators

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

TODO we need to change stateful executor to handle subgraph.

When two node entries refer to the same output of a node, we should create only one var node for these two node entries.

We can't get a list of variable names from a hashtable. The order can't be guaranteed. Python2 and Python3 output different orders.

* Test input a graph. * Update foreach to execute the subgraph. * print inputs/outputs in foreach. * Remove print. * add test code for foreach. * exec foreach outside the engine. * Implements forward of foreach. * Add support for variable numbers of inputs and outputs. * Add a python wrapper for foreach. * Fix the order of inputs. * add test with lstm. * hide C version of foreach. * fix a bug temporarily. * Test free variables. * change for the new interface of InputGraph attribute. * Add attribute to the subgraph. * Handle free variables. * Get all input symbols of a subgraph. * Fix shape, dtype and storage inference. * reorganize the output of foreach. * Add a gluon RNN unroll with symbol foreach. * print unnecessary print. * have imperative and symbolic foreach. * Fix an error after moving foreach. * Fix imperative foreach * Fix a minor problem. * Use CachedOp to execute subgraph. * update TODO. * make foreach op use FStatefulComputeEx. TODO we need to change stateful executor to handle subgraph. * Add backward. * Fix bugs. * enable backward test in lstm. * Fix a bug in foreach backward for free variables. * change for the new CachedOp. * Detect the backward computation. * Fix bugs in foreach. * fix tests. * update tests. * check state shape. * enable nested foreach. * remove print. * fix a bug in test. * handle infer storage type for backward. * address comments. * address comments. * move some common functions out. * address comments. * fix lint. * Fix lint. * add doc. * undo modification in imperative.h * add doc and remove example code. * fix lint. * fix lint. * Fix lint. * make nd.foreach and sym.foreach consistent. * fix compile error. * address comments. * update. * check for loop only works for dense arrays. * move control flow op out of nn/ * fix include. * add a test in gluon. * work for GPU. * small fix. * remove subgraph_name * create loop state for reuse in the future. * move code. * Revert "remove subgraph_name" This reverts commit 977f562. * cut graph. * rename new var nodes. * Fix tests. * Fix bugs caused by ctypes (apache#29) * Add save/load json in testcases for foreach (apache#30) * support subgraph in stateful executor. * Fix compilation. * fix a bug when a subgraph has variable nodes. * Fix a bug of getting symbols. * copy var nodes. * Fix getting op states. * fix lint error. * address comments. * fix lint error. * simplify the execution of subgraph in the main thread. * fix lint error. * avoid waiting for computation in each iteration. * reuse cached op for inference. * share memory across mini-batches. * reuse memory. reuse memory between iterations in inference. reuse memory between mini-batches in training. * add tests for multiple batches. * remove entry. * add benchmark for foreach. * benchmark large batch size. * Fix the benchmark for GPU. * address comments. * update shape/dtype/storage inference. * update contrib API docs. * support nested foreach. * use a single CachedOp for all iterations. * use large dim. * update benchmark. * update benchmark. * update benchmark. * update benchmark. * return symbol arrays correctly in MXSymbolCutSubgraph. * return symbol arrays in MXSymbolGetInputSymbols. * fix lint error. * use cachedop to infer storage in backward. * fix scala API. * update comments. * fix scala. * fix test. * fix attribute name. * move benchmark. * fix the mapping of operator inputs/outputs and subgraph inputs/outputs. * add tests for dtype/shape inference. * reorganize tests. * fix a bug of cutting NodeEntry. When two node entries refer to the same output of a node, we should create only one var node for these two node entries. * fix lint error. * handle the case that outputs are inputs. * handle the case that inputs aren't used. * handle the case without output data. * fix a bug in foreach backward. * fix a bug when there isn't output data. * Fix lint error. * test diff Gluon RNN cells. * test all symbol RNN cells. * adjust the test precision. * Fix a bug in getting a list of variable names. We can't get a list of variable names from a hashtable. The order can't be guaranteed. Python2 and Python3 output different orders. * fix lint error. * Test 1D array. * fix a bug when subgraph inputs and outputs share NDArray. * fix. * fix * add comments.

zheng-da added 30 commits July 2, 2018 18:20

Test input a graph.

e86d92c

Update foreach to execute the subgraph.

7eacfab

print inputs/outputs in foreach.

6874f7d

Remove print.

8f5e62e

add test code for foreach.

1dd35a8

exec foreach outside the engine.

1f117cb

Implements forward of foreach.

cc29fc1

Add support for variable numbers of inputs and outputs.

036aada

Add a python wrapper for foreach.

84e0e24

Fix the order of inputs.

f2a28f0

hide C version of foreach.

1c4cf0e

fix a bug temporarily.

9aa896d

add test with lstm.

6784408

Test free variables.

f488647

change for the new interface of InputGraph attribute.

d9b0c50

Add attribute to the subgraph.

1188d97

Handle free variables.

c0cd6ac

Get all input symbols of a subgraph.

74d280b

Fix shape, dtype and storage inference.

2bc80e3

reorganize the output of foreach.

68faa17

Add a gluon RNN unroll with symbol foreach.

3751ca7

print unnecessary print.

b98e06d

have imperative and symbolic foreach.

fc575fe

Fix an error after moving foreach.

37da6fb

Fix imperative foreach

f41235c

Fix a minor problem.

214c1c2

Use CachedOp to execute subgraph.

9aabc74

update TODO.

7fc0155

make foreach op use FStatefulComputeEx.

0d3613a

TODO we need to change stateful executor to handle subgraph.

Add backward.

f33d0f4

zheng-da added 23 commits July 2, 2018 18:20

fix attribute name.

64e4ff6

move benchmark.

d243c12

fix the mapping of operator inputs/outputs and subgraph inputs/outputs.

3afc4d4

add tests for dtype/shape inference.

62901fe

reorganize tests.

14b8fb9

fix a bug of cutting NodeEntry.

f7d7f17

When two node entries refer to the same output of a node, we should create only one var node for these two node entries.

fix lint error.

7d012d9

handle the case that outputs are inputs.

b83253d

handle the case that inputs aren't used.

275bbf1

handle the case without output data.

0e6df9a

fix a bug in foreach backward.

dfadc8d

fix a bug when there isn't output data.

5e9cf5f

Fix lint error.

696f53c

test diff Gluon RNN cells.

094977d

test all symbol RNN cells.

7b016ae

adjust the test precision.

9609ce8

Fix a bug in getting a list of variable names.

fa8abbd

We can't get a list of variable names from a hashtable. The order can't be guaranteed. Python2 and Python3 output different orders.

fix lint error.

8e74d80

Test 1D array.

2439105

fix a bug when subgraph inputs and outputs share NDArray.

53cdbfa

fix.

9bff317

fix

d3687ef

add comments.

392a7e4

zheng-da requested review from nswamy, szha and yzhliu as code owners July 2, 2018 18:44

piiswrong merged commit 030fbc3 into apache:master Jul 2, 2018

junrushao mentioned this pull request Jul 17, 2018

[MXNET-684] Add cond operator #11760

Merged

10 tasks

zheng-da deleted the foreach1 branch September 29, 2018 21:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MXNET-432] Add Foreach #11531

[MXNET-432] Add Foreach #11531

zheng-da commented Jul 2, 2018 •

edited

Loading

[MXNET-432] Add Foreach #11531

[MXNET-432] Add Foreach #11531

Conversation

zheng-da commented Jul 2, 2018 • edited Loading

Description

Checklist

Essentials

zheng-da commented Jul 2, 2018 •

edited

Loading