Skip to content
This repository has been archived by the owner on Feb 26, 2023. It is now read-only.

[WIP] Add Attention module #66

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

Conversation

lkhphuc
Copy link
Contributor

@lkhphuc lkhphuc commented Mar 17, 2022

Adding attention module as a wrapper around flax.linen.attention.

I think the wrapper is correct, but I can not get the test_equivalance to pass if using Initializer that need rng. I think there's some mismatch between the next_key() and my manual emulation of it.

Todo:

  • Pass test initialization with stochastic init.
  • Pass test module apply with dropout rng.
  • Add SelfAttention wrapper.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant