Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[layer] Memory optimization for backwarding of embedding layer #2289

Open
skykongkong8 opened this issue Aug 29, 2023 · 1 comment
Open

[layer] Memory optimization for backwarding of embedding layer #2289

skykongkong8 opened this issue Aug 29, 2023 · 1 comment

Comments

@skykongkong8
Copy link
Member

skykongkong8 commented Aug 29, 2023

Unlike other layers, embedding layer uses specific form of Tensor called IndexedSlices and this data contains the specific indicies of the Tensor that we are interested in.
Thus, during the backwarding process, we do not have to set all the value-tensor-shaped gradient in VarGrad, but we can optimize it by formulating the specific part for the Gradient Tensor.
In current NNTrainer code, there is no such consideration like above, but it use same-shaped but zero-filled Tensor for un-interested-indexed portion of the Tensor. (redundant size Tensor declaration)
As far as I am concerned, we should work on this part in the near future for the memory optimization purpose.

@taos-ci
Copy link
Collaborator

taos-ci commented Aug 29, 2023

:octocat: cibot: Thank you for posting issue #2289. The person in charge will reply soon.

@skykongkong8 skykongkong8 changed the title [layer] backwarding of embedding layer [layer] Memory optimization for backwarding of embedding layer Aug 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants