Expert Attention Fixes: #350
Conversation
![copybara-service[bot]](https://avatars.githubusercontent.com/in/44061?s=60&v=4){kind=small}
|
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
copybara-service
bot
force-pushed
the
test_390808726
branch
from
84b2094
to
2456b1b
Compare
|
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
- Allow moe.py to work with a tensor of "memory_length" dimension - Fix Experts Attention bug in moe.py where it would break during decoding if the input dimension was different than the output dimension. - Fix bug in ExpertsEncDecAttention where it was only doing Self-Attention on the decoder side. - Factorize expert_computation code to easily allow for using different query and memory antecedents PiperOrigin-RevId: 395259569
copybara-service
bot
force-pushed
the
test_390808726
branch
from
2456b1b
to
69bd9c7
Compare
|
We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. ℹ️ Googlers: Go here for more info. |
Expert Attention Fixes: