net design with NetBuilder #2598

Superjomn · 2017-06-26T03:47:05Z

A network design with both protobuf and AddOp support.

jacquesqiao · 2017-06-26T06:23:10Z

paddle/framework/net_design.md

+# Network Design
+
+`Network` is the container and controller of a set of operators,
+users can build a real network from a `NetDef` in protobuf message 


Should give a simple NetDef proto message

QiJune · 2017-06-27T06:44:47Z

paddle/framework/net_design.md

+  // will run.
+  virtual bool Run(Scope *scope, OpIndex begin = -1,
+                   OpIndex end = -1) const = 0;
+


As discussion in Operator Design, Run method in Operator will have a context parameter. So, I think the Run method in NetBase may also need a context parameter. Or, we can pass the context to operator run inside this method.

Actually, the NetBase Run method is what an engine should do. The engine will get variables from scope, and get runtime resources from context, and then execute.
The runtime resources may include cuda stream, cublasHandle, cudnnHandle.

I will add OpContext as NetBase.Run's argument. @QiJune
A NaiveExecutor may be necessary for the future implementation.

QiJune · 2017-06-27T06:50:04Z

paddle/framework/net_design.md

+```c++
+// create an empty scope located on CPU device.
+Scope scope(CPUPlace());
+


Maybe we also need to create a context related with a device.

Scope scope(CPUPlace()); Context context(CPUPlace()); ... net->Run(&Scope, &context);

In the offline discussion, we decide to create a context inside the net run method. Then, the context will pass to every operator's run method.

QiJune · 2017-06-27T07:29:27Z

paddle/framework/net_design.md

+  // scope will be used instead.
+  virtual bool Run(Scope *scope = nullptr, OpIndex begin = -1,
+                   OpIndex end = -1) const override;
+


A possible implementation coule be:

virtual bool Run(Scope* scope = nullptr, OpIndex begin = -1, OpIndex end = -1) const override { Context* c = CreateContext(CPUPlace()); for (auto op in ops_) { op->Run(c); } }

xinghai-sun · 2017-06-27T10:27:24Z

paddle/framework/net_design.md

+# Network Design
+
+`Network` is the container and controller of a set of operators,
+users can build a real network from a `NetDef` which is a protobuf message 


, users can --> . Users can

xinghai-sun · 2017-06-27T10:31:58Z

paddle/framework/net_design.md

+// The minimum a network should be implemented.
+class NetBase {
+ public:
+  // run all the operators and return success(true) or not, all the


run --> Run

or not --> or not (false)

", all the variables are located" --> ", with all the variables located" . 不要用逗号连接两个完整的句子，除非有合适的连词。

xinghai-sun · 2017-06-27T10:37:20Z

paddle/framework/net_design.md

+  // variables are located in `scope`. `ctx` describes the detail execution
+  // environment for ops. `begin` and `end` specify the scope of `ops_` to run,
+  // If no positive indexes are provided, all operators in `ops_` will run.
+  virtual bool Run(Scope *scope, OpContext *ctx, OpIndex begin = -1,


Since the word "context" is not too long, why not use context instead of ctx for a better readability?

xinghai-sun · 2017-06-27T10:54:58Z

paddle/framework/net_design.md

+
+  // Infer the shapes of variables required by operators in the network. The
+  // `scope` will be mutated according to the inferred shapes.
+  virtual bool InferShape(Scope *scope) = 0;


The name "InferShape" is a little ambiguous, since it does not return a shape in fact.
"CheckVariableShape" or "ValidateVariableShape" or "CheckShape" or "ValidateShape" ?

Or we can remove this function? Each operator has its own InferShape', just call op.InferShapeinBaseNet.AddOp. Make the AddOpthe only method to add an operator to a network, so thatop.InferShapewill be called every time automatically, no need to call BaseNet.InferShape` manually.

InferShape may invoke in each mini-batch run. Because the input shape will be changed in each mini-batch.

xinghai-sun · 2017-06-27T11:04:25Z

paddle/framework/net_design.md

+```
+
+Network is designed as the container of operators, to make it more extendable,
+we decoupling it from the related variable resources. 


“, to make it" --> ". To make it"

"we decoupling it" --> "we decouple it"

xinghai-sun · 2017-06-27T11:06:16Z

paddle/framework/net_design.md

+
+`Run(Scope* scope)` takes the scope as a argument so that it can run in different scopes.
+
+Finally, `NetBase` can be used as followed


as followed --> as follows

xinghai-sun · 2017-06-27T11:08:05Z

paddle/framework/net_design.md

+
+## `PlainNet` as a simple implementation of `BaseNet`
+
+A very basic implementation is as followed, all it does is simply to run every operators in sequence.


"as followed, all it" --> "as follows. All it"

xinghai-sun · 2017-06-27T11:12:51Z

paddle/framework/net_design.md

+  // Add a operator which is identified as `type` and has attributes described
+  // in `attrs`, the `inputs` are the keys of readonly input variables,
+  // `outputs` are keys of mutable output variables. An `OpIndex` will be
+  // returned which indicates the offset of the new operator in `ops_`.


which indicates --> to indicate

xinghai-sun · 2017-06-27T11:24:45Z

paddle/framework/net_design.md

+
+## Compatibility with RNN
+
+Benefit from the decoupling of `PlainNet.Run` and `Scope`, `PlainNet` is compatible with future RNN design, 


Benefit --> Benefitting

wangkuiyi · 2017-06-29T03:33:17Z

paddle/framework/net_design.md

@@ -0,0 +1,251 @@
+# Network Design
+
+`Network` is the container and controller of a set of operators,


controller => execution engine

Will the framework have an independent Executor?
or merge the Executor into `Network'? @wangkuiyi

wangkuiyi · 2017-06-29T03:34:41Z

paddle/framework/net_design.md

+user can build a real network from a `NetDef` which is a protobuf message 
+and use `Network.Run()` to run all the operators in the network.
+
+The `Network` will


A network object knows all Operators belonging to this network. Variables, which are inputs and outputs of these operators, are created and managed by a hierarchy of Scope objects.

wangkuiyi · 2017-06-29T05:50:12Z

paddle/framework/net_design.md

+typedef int OpIndex;
+
+// The minimum a network should be implemented.
+class NetBase {


NetBase ==> Net, because our other base classes, including Operator, don't have the Base suffix. Let's unify the style.

wangkuiyi · 2017-06-29T05:51:02Z

paddle/framework/net_design.md

+  virtual OpIndex AddOp(const proto::OpDef &def) = 0;
+
+  // Add optimizer operators acctording to `attrs`.
+  virtual void AddOptimizerOps(const OptAttrs &attrs) = 0;


What is an "optimizer op"? What's the difference between "optimzier op" and "op" as in AddOp?

AddOp add normal forward operators and is called by user or network's constructor.
OptimizerOp is the optimizer (adam for example) on model parameter, network will scan all the forward operators and get their parameters, then add optimizer operators for each parameter.

User can access the AddOp function, but AddOptimizerOps should be called by the network after all the forward operators are added.

wangkuiyi · 2017-06-29T05:51:55Z

paddle/framework/net_design.md

+  // variables are located in `scope`. `context` describes the detail execution
+  // environment for ops. `begin` and `end` specify the scope of `ops_` to run,
+  // If no positive indexes are provided, all operators in `ops_` will run.
+  virtual bool Run(Scope *scope, OpContext *context, OpIndex begin = -1,


Suppose that the current Net represent an RNN's step-net, what scope should the parameter scope points to? -- the scope of the current step-net, or its parent?

In Scope's design, there is a member std::shared_ptr<Scope> parent_ {nullptr}; ,
which is a reference to the pararent's scope.

So this scope means the current stepnet's scope, network can get its pararent's scope by access its parent_.

reference Scope Design

wangkuiyi · 2017-06-29T05:53:09Z

paddle/framework/net_design.md

+class PlainNet final : public NetBase {
+ public:
+  // Create a network describe by `def`.  NetDef is the definition of a network.
+  PlainNet(const NetDef &def);


NetDef ==> NetDesc

wangkuiyi · 2017-06-29T05:54:03Z

paddle/framework/net_design.md

+
+ protected:
+  // Create operators accordding to `def`, will be called by the constructor.
+  bool BuildNet(const NetDef &def);


Why BuildNet returns a bool value? How would the constructor treat it?

Maybe a Status or Error return type here.

wangkuiyi · 2017-06-29T05:55:14Z

paddle/framework/net_design.md

+
+```c++
+// copy some `vars` form `source` to `target`
+void Copy(const Scope &source, Scope &target,


I don't see how should we define this function. Can you please complete it so to make this example comprehensive?

just several lines of pseudo code here, maybe I should remove this RNN demo.

wangkuiyi · 2017-06-29T06:01:05Z

paddle/framework/net_design.md

+for (int i = 0; i < num_states + 1; i++) {
+  // Initialize all rnn state scopes, copy parameters and so on.
+  rnn_states[i].CreateVars(rnn_step_net_def);
+  Copy(default_scope, rnn_states[i], rnn_related_vars);


How comes this rnn_related_vars?

this is a pseudo code, this means RNN related variables' string keys, rnn_related_vars should be set during RNN step net's construction.

wangkuiyi · 2017-06-29T06:02:08Z

paddle/framework/net_design.md

+## Compatibility with RNN
+
+Benefitting from the decoupling of `PlainNet.Run` and `Scope`, `PlainNet` is compatible with future RNN design, 
+for example we can implement a simple recurrent neural network as follows


Where is the following code snippet supposed to be? In the constructor of RNNOp?

Just a demo code snippet, to make sure the net is compatible with Scope in RNN's implementation.

wangkuiyi

I approved this PR, because it provides a general skeleton of Net. We can have futher discussions and polish in coming PRs.

@Superjom : I'd suggest that we verify the design with some "real" examples, other than pseudo-code snippets, in following PRs.

Superjomn added 24 commits June 22, 2017 09:57

move net_design to framework

5886959

add nullptr to Network.Run

a00900a

add usage

4fb581b

remove scope* in construction

bb33f7a

remove AddOp from user's interface

a7dbfe0

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into develop

8cf2d60

net new design

2933da0

add more details to net design

7d5b1b2

disable CreateOp from user

b6450da

move w1 to scope

a46506d

add rnn compatibility

3c7bf55

fix english grammer

ebe143d

rrn -> rnn

4e5a359

change CreateNet result to unique_ptr

9be4c74

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into develop

7102b80

make Run() stateless

2953215

complete a unique_ptr's >

58fdcd0

pr details fix

417b279

add InferShape

b68c8bf

rename "ScratchNet" -> "PlainNet"

62823b1

Merge branch 'develop' of github.com:PaddlePaddle/Paddle into develop

deff7cc

detail config

740f3a6

add three methods to NetBase

e43eec8

add NetBuilder

e65750b

Superjomn requested review from reyoung, jacquesqiao, dzhwinter, wanghaoshuang, wangkuiyi and xinghai-sun June 26, 2017 03:48

Superjomn mentioned this pull request Jun 26, 2017

move net_design to framework #2553

Closed

jacquesqiao reviewed Jun 26, 2017

View reviewed changes

add InferShape to NetBuilder.Run

4fa6144

Superjomn assigned Superjomn, dzhwinter, xinghai-sun and wanghaoshuang Jun 27, 2017

rename ApplyGradient, ApplyOptimizer -> AddGradientOps, AddOptimiz

9436880

QiJune reviewed Jun 27, 2017

View reviewed changes

add OpContext to NetBase.Run

af67cd6

xinghai-sun reviewed Jun 27, 2017

View reviewed changes

Superjomn added 4 commits June 28, 2017 08:49

merge xinhai's review

bcc885f

remove InferShape

f18fa48

rename PlainNet::CreateNet -> BuildNet

0c7ccea

add InferShape comments

8ae4566

wangkuiyi reviewed Jun 29, 2017

View reviewed changes

add Error and other rename actions

7a0d4ba

wangkuiyi approved these changes Jun 29, 2017

View reviewed changes

Superjomn merged commit 06156da into PaddlePaddle:develop Jul 4, 2017


		`Run(Scope* scope)` takes the scope as a argument so that it can run in different scopes.

		Finally, `NetBase` can be used as followed


		## `PlainNet` as a simple implementation of `BaseNet`

		A very basic implementation is as followed, all it does is simply to run every operators in sequence.


		## Compatibility with RNN

		Benefit from the decoupling of `PlainNet.Run` and `Scope`, `PlainNet` is compatible with future RNN design,

		@@ -0,0 +1,251 @@
		# Network Design

		`Network` is the container and controller of a set of operators,

net design with NetBuilder #2598

net design with NetBuilder #2598

Conversation

Superjomn commented Jun 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

QiJune Jun 27, 2017 • edited

Choose a reason for hiding this comment

Superjomn Jun 27, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

QiJune Jun 27, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Superjomn Jun 28, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Superjomn Jun 29, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Superjomn Jun 29, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Superjomn Jun 29, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangkuiyi left a comment

Choose a reason for hiding this comment

QiJune Jun 27, 2017 •

edited

Superjomn Jun 27, 2017 •

edited

QiJune Jun 27, 2017 •

edited

Superjomn Jun 28, 2017 •

edited

Superjomn Jun 29, 2017 •

edited

Superjomn Jun 29, 2017 •

edited

Superjomn Jun 29, 2017 •

edited