-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add alpha_dropout in nn.functional and nn.layer, test=develop #26365
add alpha_dropout in nn.functional and nn.layer, test=develop #26365
Conversation
Thanks for your contribution! |
keep_mask) | ||
|
||
#apply mask | ||
alpha_p = layers.fill_constant(shape=[1], dtype=dtype, value=alpha_p) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这些 tensor * scalar 的情形其实用 scale 可能会更快。
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
python/paddle/nn/layer/common.py
Outdated
|
||
class AlphaDropout(layers.Layer): | ||
""" | ||
:alias_main: paddle.nn.AlphaDropout |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
alis 可以不用写了。由官网解析自动处理
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
python/paddle/nn/layer/common.py
Outdated
[Self-Normalizing Neural Networks](https://arxiv.org/abs/1706.02515) | ||
`paddle.nn.functional.alpha_dropout` | ||
|
||
In dygraph mode, please use ``eval()`` to indicate whether it is in test phrase or not. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
to switch to evaluation mode, where dropout is disabled.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@@ -456,6 +460,80 @@ def _is_list_or_turple_(data): | |||
return out | |||
|
|||
|
|||
def alpha_dropout(x, p=0.5, training=True, name=None): | |||
""" | |||
alpha_dropout function. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
delete this line
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
x (Tensor): The input tensor. The data type is float32 or float64. | ||
p (float): Probability of setting units to zero. Default 0.5. | ||
training (bool): A flag indicating whether it is in train phrase or not. Default True. | ||
name (str|None): A name for this layer(optional). If set None, the layer |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
use template phrase for name.
name (str|None): A name for this layer(optional). If set None, the layer | ||
will be named automatically. | ||
Returns: | ||
A Tensor representing the dropout, has same shape and data type with `x`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
with -> as
x = np.array([[-1, 1], [-1, 1]]).astype('float32') | ||
x = paddle.to_tensor(x) | ||
y_train = paddle.nn.functional.alpha_dropout(x, 0.5) | ||
y_test = paddle.nn.functional.alpha_dropout(x, 0.5, training=False) #test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove #test
# [[-0.10721093, 1.6655989 ], [-0.7791938, -0.7791938]] (randomly) | ||
print(y_test.numpy()) | ||
""" | ||
if not isinstance(p, (float, int)): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
docstring only says float
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
python/paddle/nn/layer/common.py
Outdated
def __init__(self, p=0.5, name=None): | ||
super(AlphaDropout, self).__init__() | ||
self.p = p | ||
self.training = _dygraph_tracer()._train_mode |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Layer
now has train()
/eval()
, this can be removed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
add alpha_dropout and AlphaDropout; |
@@ -80,42 +80,34 @@ def interpolate(input, | |||
'trilinear' : Trilinear interpolation | |||
'nearest' : Nearest neighbor interpolation | |||
'bicubic' : Bicubic interpolation | |||
Linear interpolation is the method of using a line connecting two known quantities |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
unrelated to this PR?
For details of linear interpolation, please refer to Wikipedia: | ||
https://en.wikipedia.org/wiki/Linear_interpolation. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
auto formatting?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
[4 0 6]] | ||
(3) What about ``axis=[0, 1]`` ? This means the dropout is performed in all axes of x, | ||
which is the same case as default setting ``axis=None`` . | ||
(4) You may note that logically `axis=None` means the dropout is performed in no axis of x, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no axis
-> all axis
When x is a 4d tensor with shape `NCHW`, we can set ``axis=[0,1]`` and the dropout will be performed | ||
in channel `N` and `C`, `H` and `W` is tied, i.e. | ||
paddle.nn.dropout(x, p, axis=[0,1]) | ||
This is something we called dropout2d. Please refer to ``paddle.nn.functional.dropout2d`` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove This is something we called dropout2d.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
PR types
New features
PR changes
APIs
Describe
add apis of alpha_dropout:
(1) paddle.nn.functional.alpha_dropout(x, p=0.5, training=True, name = None)
(2) paddle.nn.AlphaDropout(p=0.5, name = None)