add external memory network demo #696
Conversation
Haichao-Zhang
force-pushed
the
ext_mem_demo
branch
2 times, most recently
from
e23f54e
to
1dddec7
Compare
Haichao-Zhang
force-pushed
the
ext_mem_demo
branch
from
1dddec7
to
9b4ea69
Compare
Haichao-Zhang
changed the title
| self.name = name | ||
| self.mem_slot_size = mem_slot_size | ||
| self.mem_fea_size = mem_fea_size | ||
| self.scale = 5 |
There was a problem hiding this comment.
This has been corrected.
| self.scale = 5 | ||
| self.external_memory = memory(name=self.name, | ||
| size=mem_fea_size*mem_slot_size, | ||
| boot_bias= ParamAttr(initial_std=0.01, |
There was a problem hiding this comment.
This has been updated.
| bias_attr = False, | ||
| act = SoftmaxActivation(), | ||
| size = self.mem_slot_size, | ||
| name='read_weight') |
There was a problem hiding this comment.
in order to avoid name confict when using multiple memory, this and other names should be prefixed by self.name
There was a problem hiding this comment.
Similar issues have been addressed.
|
|
||
| return memory_output | ||
|
|
||
| def MakeConstantVector(self, vec_size, value, dummy_input): |
There was a problem hiding this comment.
python naming convention: make_constant_vector
There was a problem hiding this comment.
Changed the function name following the convention.
| memory_removed = mixed_layer(input = [identity_projection(input=self.external_memory), | ||
| identity_projection(input=memory_remove_neg)], | ||
| bias_attr = False, | ||
| act = LinearActivation()) |
There was a problem hiding this comment.
78 and 81 can be combinded as written as: memory_removed = self.external_memory - memory_remove.
See https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/trainer_config_helpers/tests/configs/math_ops.py
There was a problem hiding this comment.
This part of the code has been updated using math_ops
| print_layer(input=[erase_vec]) | ||
| print_layer(input=[add_vec]) | ||
|
|
||
| out_prod = out_prod_layer(norm_cosine_similarity_write, erase_vec, name="outer") |
There was a problem hiding this comment.
Creating a constant vector erase_vec for this is very ugly. A nicer way to do this is to enhance "repeat" layer to allow repeat in both directions, similar to "repmat" in matlat.
There was a problem hiding this comment.
Looking into the repeat layer currently.
|
|
||
| out_prod_add = out_prod_layer(norm_cosine_similarity_write, add_vec, name="outer_add") | ||
|
|
||
| memory_output = mixed_layer(input = [identity_projection(input=memory_removed), |
There was a problem hiding this comment.
Using addto_layer can make this looks simpler.
There was a problem hiding this comment.
Switch to addto_layer
| from paddle.trainer_config_helpers import * | ||
|
|
||
|
|
||
| class ExternalMemory(object): |
There was a problem hiding this comment.
need to comment for the class, and its member functions.
There was a problem hiding this comment.
Comments have been added to both the class and member functions.
Haichao-Zhang
force-pushed
the
ext_mem_demo
branch
from
eaa7d62
to
1be184e
Compare
|
感谢您给PaddlePaddle贡献代码。由于Paddle V1/V2版本已不再维护，相关代码也已从develop分支上删除，因此关闭您的PR，欢迎您向Paddle最新版-Fluid贡献代码。 |
) * add api_guides low_level backward parameter program_en * Apply suggestions from code review Co-Authored-By: zy0531 <48094155+zy0531@users.noreply.github.com> * Apply suggestions from code review Co-Authored-By: zy0531 <48094155+zy0531@users.noreply.github.com> * Update backward_en.rst * Update parameter_en.rst * Update program_en.rst * Update doc/fluid/api_guides/low_level/program_en.rst
This PR includes an example implementation of an external memory network and example usage with a simple task.