Question: How to reduce the memory in this project

[yangyaofei]

Hi, I read your paper ,it's great. I'm very interesting about how to reduce the memory in the real project.

I guess the memory things are:
in

adaptive-span/adaptive_span.py

Line 127 in d882404

key_pe = key_pe[:, :, trim_len:]

But I just see you cut the key_pe and It's just reduce a little memory and wouldn't help for reduce the Q K things I think.

So. can you explain How to reduce the memory in the code?

thanks

[connection-ai]

I think that unskewing attention probs matrix (modeling.py line 73) can reduce the memory a lot.

[yangyaofei]

I will look that again , thank you dalpo814 <notifications@github.com> 于 2019年7月16日周二 16:55写道：

…

I think that unskewing attention probs matrix (modeling.py line 73) can reduce the memory a lot. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7?email_source=notifications&email_token=AB4RBEQ5H7T5P64MBUXELA3P7WEIRA5CNFSM4IB572B2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2AFX5A#issuecomment-511728628>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB4RBEQVD66ICUY4GKYRX43P7WEIRANCNFSM4IB572BQ> .

[tesatory]

K is being cut too here

adaptive-span/adaptive_span.py

Line 123 in d882404

key = F.pad(key, [0, 0, -trim_len_cache, 0])

[yangyaofei]

@tesatory sorry, I can't get it . that's code will pad the key to zero but not reduce the memory.

[tesatory]

Sorry wrong line. It is cut here

adaptive-span/adaptive_span.py

Line 118 in d882404

key = key[:, trim_len_cache:, :]

[yangyaofei]

@tesatory thank you to reply, I still have some confuse.
It's something like the 0:trim_len_catch was computed, in this time it just use the catch that was computed previous .

I want to know is :

in your paper, you said you use a sub-network to help attention to pick a range to compute the attention. It will reduce memory, because we can cut the unnecessary part. I can't find that part,because for every element in attention, the range is different. for 5th and length is 30 is from 0 to 35 like that. I have some idea same as that, but I can't find a proper way to do that.

I read your paper, I think your project can help.
thank you

[yangyaofei]

@tesatory oh, sorry, I realize your paper is not what I thought. It's reduce the memory in generation. Thanks