Very slow (exponential) parsing speed when having a large depth of addto_layer

[xyz2357]

When there's a lot of addto_layers in a network, it will takes exponential parsing time.
E.G. 1:

def whole_network(src_embedding):
    enc = src_embedding
    for i in range(depth):
        enc = addto_layer([enc, enc])

    pred = fc_layer(input=fc_layer(
                        input=enc,
                        size=dim_embedding,
                        act=ReluActivation()
                        ),
                    size=label_dict_len,
                    act=SoftmaxActivation())
    return pred

E.G. 2:

def whole_network(src_embedding):
    enc = src_embedding
    for i in range(depth):
        enc_res = fc_layer(input=enc, size=dim_embedding)
        enc_res = fc_layer(input=enc_res, size=dim_embedding)
        enc = addto_layer([enc, enc_res])

    pred = fc_layer(input=fc_layer(
                        input=enc,
                        size=dim_embedding,
                        act=ReluActivation()
                        ),
                    size=label_dict_len,
                    act=SoftmaxActivation())
    return pred

Both will costs a huge amount of time to parse (test by the nest_diagram tool).
My parsing time:

depth: 4,   parsing time: 0.16.
depth: 8,   parsing time: 0.16.
depth: 12, parsing time: 0.33.
depth: 16, parsing time: 2.02.
depth: 20, parsing time: 32.08.
depth: 21, parsing time: 67.05.
depth: 22, parsing time: 131.48.
depth: 23, parsing time: 268.82.

[wangkuiyi]

Just mark that " parsing time means the time used for parsing and generating the protobuf".

[reyoung]

This issue is fixed by #2802. It because when we parsing network topology by outputs(xxx), it just runs a depth-first search without any optimization.

In #2802, if a node is visited before, that node is just skipped. It will fix this issue.

You can just change network.py in paddle python package by this patch.

[xyz2357]

Thanks a lot!