Add unified RNN APIs #26588

iclementine · 2020-08-24T03:19:09Z

PR types

New features

PR changes

APIs

Describe

Add unified RNN APIs

RNN cells: SimpleRNNCell, LSTMCell, GRUCell
RNN networks: SimpleRNN, LSTM, GRU,
high order RNN wrapper: RNN, BiRNN

test=develop

…o rnn

paddle-bot-old · 2020-08-24T03:19:18Z

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

guoshengCS

LGTM

guoshengCS · 2020-08-26T02:26:04Z

python/paddle/nn/layer/rnn.py

+        self.is_reverse = is_reverse
+        self.time_major = time_major
+
+    def forward(self, inputs, initial_states=None, sequence_length=None):


这里是否要增加**kwargs呢

这个可以加上的，Done

guoshengCS · 2020-08-26T02:29:22Z

python/paddle/nn/layer/rnn.py

+        outputs, final_states = F.birnn(self.cell_fw, self.cell_bw, inputs,
+                                        initial_states, sequence_length,
+                                        self.time_major)
+        return outputs, final_states


F.birnn和这里返回的final_states 是否需要像outputs一样把双向的内容给concat起来呢

F.birnn 是一个比较低级的接口，它不能预设两个 cell 的层数或者 hidden_size 相等，也不能预设两个 shell 的类型相等（比如说正向是 SimpleRNN, 但是反向是一个 LSTM），所以没法这样做。

guoshengCS · 2020-08-26T02:35:55Z

python/paddle/nn/layer/rnn.py

+
+        outputs, final_states = F.birnn(self.cell_fw, self.cell_bw, inputs,
+                                        initial_states, sequence_length,
+                                        self.time_major)


**kwargs是否要加上呢

这个可以加上的，Done

swtkiwi

1、请中英文文档一起提交review
2、请附上预览图

swtkiwi · 2020-08-26T02:39:06Z

python/paddle/fluid/layers/rnn.py

+        **kwargs: Additional keyword arguments. Arguments passed to `cell.call`. 
+
+    Returns:
+        outputs (Tensor): A (possibly nested structure of) tensor variable[s],


variable可以删除

swtkiwi · 2020-08-26T02:40:53Z

python/paddle/fluid/layers/rnn.py

+            cell_bw = LSTMCell(16, 32)
+            inputs = paddle.rand((2, 23, 16))
+            outputs, final_states = paddle.nn.functional.birnn(cell_fw, cell_bw, inputs)
+


1、示例输入一般不要用随机生成，最好是具体的例子
2、注释一下具体输出内容

不用随机生成也无法保证输出。

所以含参 Layer 相关的一律不写输入输出

jzhang533 · 2020-08-26T02:00:30Z

python/paddle/nn/layer/rnn.py

+
+    Please refer to `Finding Structure in Time 
+    <https://crl.ucsd.edu/~elman/Papers/fsit.pdf>`_ for more details.
+


需要说明一下参数的默认初始化的方法。
因为RNN系列的参数初始化方法跟ParamAttr的默认的初始化方法不一样。
下同。

有何不一样？

指的是默认 Uniform(-std, std) 初始化这个事情吗？

是的呀。
ParamAttr默认是xavier吧，用户沿着这个文档找到ParamAttr的说明，会误以为rnn也是xavier初始化的。

jzhang533 · 2020-08-26T02:08:28Z

python/paddle/nn/layer/rnn.py

+
+    def forward(self, inputs, states=None):
+        r"""
+        Given the input and previous atate, compute the output and update state.


typo 'state`

jzhang533 · 2020-08-26T02:12:27Z

python/paddle/nn/layer/rnn.py

+        hidden_size (int): The hidden size.
+        nonlinearity (str): The activation in the SimpleRNN cell. It can be 
+            `tanh` or `relu`. Defaults to `tanh`.
+        weight_ih_attr(ParamAttr, optional): The parameter attribute for 


这几个参数的说明文档，直接说input to hidden weights ; hidden to hidden weights，会容易理解一些。
因为文档里也没找到哪儿再在解释weight_ih

已经加上了 Parameter section 来解释参数和公式中的符号的对应关系。

jzhang533 · 2020-08-26T02:30:50Z

python/paddle/nn/layer/rnn.py

+            `[time_steps, batch_size, ...]`. Defaults to False.
+
+    Inputs:
+        inputs (Tensor): A (possibly nested structure of) tensor variable[s]. 


"possibly nested structure of"
这句话有些含糊。

这个版本里这里就是个batch * length * input_size 的 Tensor

还是说可以结合forward时的sequence_length的参数，有另外的用法?

"possibly nested structure of" 是之前就这么设计的。

我认为支持 (T, B, C) 以及 (B, T, C) 两种布局的输入。

jzhang533 · 2020-08-26T02:33:45Z

python/paddle/nn/layer/rnn.py

+            in RNN.
+        initial_states (Tensor|list|tuple, optional): A (possibly nested structure of)
+            tensor[s], representing the initial state for the rnn cell. 
+            If not provided, `cell.get_initial_states` would be used to produce


get_initial_states的默认行为是？

全 0 初始化

jzhang533 · 2020-08-26T02:47:19Z

python/paddle/nn/layer/rnn.py

+class BiRNN(Layer):
+    r"""
+    Wrapper for bidirectional RNN. It assembles two RNN cells by performing
+    forward and backward RNN separately, and concat outputs.


第二句英语感觉让新手不容易理解。（我也没想好其他合适的说法）

Wrapper for bidirectional RNN. It takes two RNN cells as parameters and build a bidiretional RNN. A BiRNN applies forward RNN and backward RNN separately and concats the outputs along the last axis.

这样？

jzhang533 · 2020-08-26T02:50:56Z

python/paddle/nn/layer/rnn.py

+            `tanh` or `relu`. Defaults to `tanh`.
+        direction (str): The direction of the network. It can be "forward", 
+            "backward" and "bidirectional". Defaults to "forward".
+        dropout (float): The droput probability. Dropout is applied to the 


default value?

defaults to "forward".

jzhang533 · 2020-08-26T02:51:29Z

python/paddle/nn/layer/rnn.py

+            "backward" and "bidirectional". Defaults to "forward".
+        dropout (float): The droput probability. Dropout is applied to the 
+            input of each layer except for the first layer.
+        time_major (bool): Whether the first dimension of the input means the


default value

jzhang533 · 2020-08-26T02:53:10Z

python/paddle/nn/layer/rnn.py

+
+        for i, rnn_layer in enumerate(self):
+            if i > 0:
+                inputs = F.dropout(


确认一下。
这里用functional 形式的dropout的话，Layer.train和Layer.evaluate能正确处理吗？

这个我稍候确认一下 Layer 的 eval 行为以及 Program 的 clone 行为

已经加上了 training 参数

jzhang533 · 2020-08-26T02:56:16Z

tools/wlist.json

+        "RNN",
+        "BiRNN",
+        "RNNCellBase",
+        "RNNCellBase.get_initial_states"


只有base class需要加入这里吧？

文档解析工具不知出什么问题，每个都说没有 sample code

… rnn

XiaoguangHu01

LGTM

XiaoguangHu01 · 2020-08-26T11:08:08Z

python/paddle/nn/layer/rnn.py

+            if nonlinearity == "tanh" \
+            else F.relu
+
+    def forward(self, inputs, states=None):


这里inputs 是不是需要改成 input？复数表示有多个Tensor

这个看怎么理解？因为 batch 本身就已经是复数。

XiaoguangHu01 · 2020-08-26T11:10:34Z

python/paddle/nn/layer/rnn.py

+    def __init__(self,
+                 input_size,
+                 hidden_size,
+                 nonlinearity="tanh",


激活函数，建议用activation="tanh"

已经修改

XiaoguangHu01 · 2020-08-26T12:19:20Z

python/paddle/nn/layer/rnn.py

+                 input_size,
+                 hidden_size,
+                 num_layers=1,
+                 nonlinearity="tanh",


建议用activation

已经修改

XiaoguangHu01 · 2020-08-26T12:29:00Z

python/paddle/nn/layer/rnn.py

+    and mostly used in RNN.
+    """
+
+    def get_initial_states(self,


这个API是否需要提供给开发者使用？看起来可以作为一个内部的API

基类的接口，用户本身也是可以调用。

XiaoguangHu01 · 2020-08-26T12:38:08Z

python/paddle/nn/layer/rnn.py

+            prev_h = paddle.randn((4, 32))
+
+            cell = paddle.nn.LSTMCell(16, 32)
+            y, h = cell(x, prev_h)


返回值不对应

已经修改。

jzhang533

lgtm
some followup:

make explanation to "nested structure of tensors", "padded sequence", "initial states" more clear.
only Base Class could be added to the white list (for CI).

… rnn

jzhang533

lgtm

XiaoguangHu01

LGTM

guoshengCS and others added 7 commits August 10, 2020 12:33

Add RNN related apis in paddl.nn

eb81185

test=develop

Merge branch 'add-nn-rnn' of https://github.com/guoshengCS/Paddle int…

56398fd

…o rnn

new rnn api, cell almost done

4768854

add new progresses in rnn APIs for 2.0

a88fb88

merge upstream, resolve conflicts

9687311

refine rnn APIs and docstrings.

5fb65ba

fix conflicts

59a79a3

chenfeiyu added 8 commits August 24, 2020 17:37

add unittets

156b490

resolve conflicts

a029dc7

disable gpu tests when paddle is not compiled with cuda support

14574d4

resolve conflicts

18d5583

remove unnecessary imports

779e226

resolve conflicts

a6fd466

fix docstring

565ddb9

add to no_sample wlist

9569a55

iclementine force-pushed the rnn branch from 4832a4a to 9569a55 Compare August 25, 2020 04:00

chenfeiyu added 2 commits August 25, 2020 12:03

resolve conflicts

d173a3b

backport to python2 to avoid yield from

07bde98

guoshengCS previously approved these changes Aug 26, 2020

View reviewed changes

guoshengCS reviewed Aug 26, 2020

View reviewed changes

swtkiwi reviewed Aug 26, 2020

View reviewed changes

jzhang533 reviewed Aug 26, 2020

View reviewed changes

chenfeiyu added 2 commits August 26, 2020 16:12

add **kwargs, fix typos

ed3d925

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

092b0d3

… rnn

iclementine dismissed guoshengCS’s stale review via 092b0d3 August 26, 2020 08:20

update docstrings for birnn

d843db8

iclementine force-pushed the rnn branch from 64ed5ad to d843db8 Compare August 26, 2020 08:42

PaddlePaddle locked and limited conversation to collaborators Aug 26, 2020

PaddlePaddle unlocked this conversation Aug 26, 2020

XiaoguangHu01 previously approved these changes Aug 26, 2020

View reviewed changes

jzhang533 previously approved these changes Aug 27, 2020

View reviewed changes

raindrops2sea previously approved these changes Aug 27, 2020

View reviewed changes

rename argument for SimpleRNN and SimpleRNNCell, fix sample code

d0f9fba

iclementine dismissed stale reviews from raindrops2sea, jzhang533, and XiaoguangHu01 via d0f9fba August 27, 2020 06:54

chenfeiyu added 2 commits August 27, 2020 15:40

add default value for initial_states in fluid.layers.birnn

db45fa3

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

877a04a

… rnn

PaddlePaddle locked and limited conversation to collaborators Aug 27, 2020

PaddlePaddle unlocked this conversation Aug 27, 2020

raindrops2sea approved these changes Aug 27, 2020

View reviewed changes

jzhang533 approved these changes Aug 27, 2020

View reviewed changes

XiaoguangHu01 approved these changes Aug 27, 2020

View reviewed changes

iclementine merged commit f408301 into PaddlePaddle:develop Aug 27, 2020


		Please refer to `Finding Structure in Time
		<https://crl.ucsd.edu/~elman/Papers/fsit.pdf>`_ for more details.

Add unified RNN APIs #26588

Add unified RNN APIs #26588

Conversation

iclementine commented Aug 24, 2020

PR types

PR changes

Describe

paddle-bot-old bot commented Aug 24, 2020

guoshengCS left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

swtkiwi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iclementine Aug 27, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jzhang533 left a comment

Choose a reason for hiding this comment

jzhang533 left a comment

Choose a reason for hiding this comment

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

iclementine Aug 27, 2020 •

edited

Loading