Add dropout operator. #3817

xinghai-sun · 2017-09-02T16:32:05Z

Resolve #3816

xinghai-sun · 2017-09-02T16:43:41Z

Dropout behaves differently in train and test mode: e.g. in the test mode, the dropout is often turned off. How can the dropout operator know whether it's in the test mode or the train mode?

qingqing01 · 2017-09-04T11:49:31Z

paddle/operators/dropout_op.cc

+  DropoutOpMaker(framework::OpProto *proto,
+                 framework::OpAttrChecker *op_checker)
+      : OpProtoAndCheckerMaker(proto, op_checker) {
+    AddAttr<float>("dropout_prob", "Dropout probability.").SetDefault(.5f);


need more detail comments, 是以概率probability置0，还是以(1-probability)置0，这块需要解释清楚。

qingqing01 · 2017-09-04T11:50:37Z

paddle/operators/dropout_op.cc

+    PADDLE_ENFORCE_NOT_NULL(ctx.InputVar("X"), "Input(X) must not be null.");
+    auto dims = ctx.Input<Tensor>("X")->dims();
+    ctx.Output<Tensor>("Out")->Resize(dims);
+    ctx.Output<Tensor>("Mask")->Resize(dims);


需要检查属性context.GetAttr<float>("dropout_prob"); 大于0

qingqing01 · 2017-09-04T11:51:21Z

paddle/operators/dropout_op.h

+        y_data[i] = 0;
+      } else {
+        mask_data[i] = 1;
+        y_data[i] = (1 - dropout_prob) * x_data[i];


对于training应该是：y_data[i] =x_data[i]

qingqing01 · 2017-09-04T11:51:51Z

paddle/operators/dropout_op.h

+    const T* x_data = x->data<T>();
+
+    float dropout_prob = context.op_.GetAttr<float>("dropout_prob");
+    int seed = context.op_.GetAttr<int>("seed");


context.op_.GetAttr -> context.GetAttr

qingqing01 · 2017-09-04T11:53:20Z

paddle/operators/dropout_op.h

+    y->mutable_data<T>(context.GetPlace());
+
+    float dropout_prob = context.op_.GetAttr<float>("dropout_prob");
+    int seed = context.op_.GetAttr<int>("seed");


context.op_.GetAttr -> context.GetAttr

qingqing01 · 2017-09-05T04:01:47Z

paddle/operators/dropout_op.h

+template <typename T>
+struct MaskGenerator {
+  float dropout_prob_;
+  int seed_;


There is no _ for the data member of struct

https://google.github.io/styleguide/cppguide.html#Variable_Names

pkuyym · 2017-09-05T03:50:17Z

paddle/operators/dropout_op.h

+template <typename T>
+struct MaskGenerator {
+  float dropout_prob_;
+  int seed_;


please follow the code styles, dropout_prob_ --> dropout_prob, seed_ --> seed

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into dropout.

qingqing01 · 2017-09-07T06:15:39Z

paddle/operators/dropout_op.cc

+  DropoutOpMaker(framework::OpProto *proto,
+                 framework::OpAttrChecker *op_checker)
+      : OpProtoAndCheckerMaker(proto, op_checker) {
+    AddAttr<float>("dropout_prob", "Probability for dropping out units.")


The template type is needed for attr.

template <typename AttrType> class DropoutOpMaker : public framework::OpProtoAndCheckerMaker { public: DropoutOpMaker(framework::OpProto *proto, framework::OpAttrChecker *op_checker) : OpProtoAndCheckerMaker(proto, op_checker) { AddAttr<AttrType>("dropout_prob", "Probability for dropping out units.")

qingqing01 · 2017-09-07T06:20:46Z

paddle/operators/dropout_op.h

+    T* y_data = y->mutable_data<T>(context.GetPlace());
+    const T* x_data = x->data<T>();
+
+    float dropout_prob = context.GetAttr<float>("dropout_prob");


template type is needed for attr.

As discussed above.

qingqing01 · 2017-09-07T06:32:13Z

paddle/operators/dropout_op.h

+    T* mask_data = mask->mutable_data<T>(context.GetPlace());
+    thrust::transform(index_sequence_begin, index_sequence_begin + size,
+                      thrust::device_ptr<T>(mask_data),
+                      MaskGenerator<T>(dropout_prob, seed));


Maybe the CPUKernel can be implemented with same way, std::transform can be used.

qingqing01 · 2017-09-07T06:34:25Z

paddle/operators/dropout_op.h

+    auto dims = grad_x->dims();
+    int size = static_cast<int>(framework::product(dims));
+    auto new_dims = framework::make_ddim({dims[0], size / dims[0]});
+    auto M = EigenMatrix<T>::From(*mask, new_dims);


EigenMatrix<int>

I think any of int, float, or T for mask data type is OK. T might be better because it avoids implicit type conversion when mask is multiplied with X.

When considering memory usage, it would be better to use a bool (single byte) or float16 (2 bytes), but that will incur the following error:

129: .terminate called after throwing an instance of 'paddle::platform::EnforceNotMet' 129: what(): This type of tensor cannot be expose to Python at [/home/work/sunxinghai/git/Paddle/paddle/pybind/tensor_py.h:36]

What do you think?

Sorry, my comment is wrong. Just use type T. Since the tensor in pybind doest not support to set and get tensor with bool type, the error for bool type is caused.

qingqing01 · 2017-09-14T12:43:16Z

paddle/operators/dropout_op.cc

+    // resize
+    auto dims = ctx.Input<Tensor>("X")->dims();
+    ctx.Output<Tensor>("Out")->Resize(dims);
+    ctx.Output<Tensor>("Mask")->Resize(dims);


LoDTensor合入后， InferShape里的Output需要改成：Output< framework::LoDTensor>

qingqing01 · 2017-09-14T12:43:34Z

paddle/operators/dropout_op.cc

+    PADDLE_ENFORCE_EQ(x_dims, mask_dims,
+                      "Dimensions of Input(X) and Mask must be the same.");
+    // resize
+    auto *x_grad = ctx.Output<Tensor>(framework::GradVarName("X"));


LoDTensor合入后， InferShape里的Output需要改成：Output< framework::LoDTensor>

qingqing01 · 2017-09-14T12:44:11Z

python/paddle/v2/framework/tests/test_dropout_op.py

+import unittest
+import numpy as np
+from gradient_checker import GradientChecker, create_op
+from op_test_util import OpTestMeta


需要使用新的单测框架

Change type of dropout_prob to template typename.

xinghai-sun added 3 commits September 2, 2017 18:33

Add dropout operator.

9a44f3d

Fixed SEGFAULT of dropout operator in GPU.

b1a1855

Correct some typos.

c657537

qingqing01 requested review from JiayiFeng, Superjomn, luotao1 and reyoung September 4, 2017 02:05

qingqing01 reviewed Sep 4, 2017

View reviewed changes

qingqing01 reviewed Sep 5, 2017

View reviewed changes

pkuyym reviewed Sep 5, 2017

View reviewed changes

qingqing01 added the OpPorting label Sep 5, 2017

xinghai-sun added 2 commits September 6, 2017 20:25

Update by following reviewers' comments.

963a4f3

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into dropout.

Correct typos for dropout operator.

4eadfc0

qingqing01 reviewed Sep 7, 2017

View reviewed changes

qingqing01 previously approved these changes Sep 11, 2017

View reviewed changes

qingqing01 added this to Doing in Port Operators Sep 14, 2017

qingqing01 reviewed Sep 14, 2017

View reviewed changes

xinghai-sun added 2 commits September 16, 2017 21:16

Merge branch 'develop' into dropout

0532662

Move dropout gpu kernel to dropout_op.cu.

32645b5

xinghai-sun dismissed qingqing01’s stale review via 32645b5 September 16, 2017 14:02

xinghai-sun added 2 commits September 19, 2017 01:44

Add is_training attr and testing phrase compuation to dropout operator.

585d12a

Change type of dropout_prob to template typename.

Merge branch 'develop' into dropout

a2798ff

qingqing01 previously approved these changes Sep 19, 2017

View reviewed changes

Remove unnecessary mask operations in test phase for dropout operator.

ffeeef8

xinghai-sun dismissed qingqing01’s stale review via ffeeef8 September 19, 2017 07:16

qingqing01 approved these changes Sep 19, 2017

View reviewed changes

xinghai-sun merged commit c7f91a9 into PaddlePaddle:develop Sep 19, 2017

xinghai-sun deleted the dropout branch September 19, 2017 09:31

qingqing01 moved this from Doing to Done in Port Operators Sep 20, 2017

heavengate pushed a commit to heavengate/Paddle that referenced this pull request Aug 16, 2021

s2anet update (PaddlePaddle#3817)

99c63ae

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add dropout operator. #3817

Add dropout operator. #3817

xinghai-sun commented Sep 2, 2017

xinghai-sun commented Sep 2, 2017

qingqing01 Sep 4, 2017

xinghai-sun Sep 6, 2017

qingqing01 Sep 4, 2017

xinghai-sun Sep 6, 2017

qingqing01 Sep 4, 2017

xinghai-sun Sep 6, 2017

qingqing01 Sep 4, 2017

xinghai-sun Sep 6, 2017

qingqing01 Sep 4, 2017

xinghai-sun Sep 6, 2017

qingqing01 Sep 5, 2017

xinghai-sun Sep 6, 2017

pkuyym Sep 5, 2017

qingqing01 Sep 7, 2017

xinghai-sun Sep 16, 2017 •

edited

qingqing01 Sep 7, 2017

xinghai-sun Sep 16, 2017

qingqing01 Sep 7, 2017

qingqing01 Sep 7, 2017

xinghai-sun Sep 16, 2017

qingqing01 Sep 18, 2017

qingqing01 Sep 14, 2017

xinghai-sun Sep 16, 2017

qingqing01 Sep 14, 2017

xinghai-sun Sep 16, 2017

qingqing01 Sep 14, 2017

xinghai-sun Sep 16, 2017

Add dropout operator. #3817

Add dropout operator. #3817

Conversation

xinghai-sun commented Sep 2, 2017

xinghai-sun commented Sep 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xinghai-sun Sep 16, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xinghai-sun Sep 16, 2017 •

edited