Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor(ivy, torch-frontend): Extends ivy.multi_head_attention and shortens torch_frontend.multi_head_attention_forward #23131

Merged
merged 84 commits into from
Sep 27, 2023
Merged

refactor(ivy, torch-frontend): Extends ivy.multi_head_attention and shortens torch_frontend.multi_head_attention_forward #23131

merged 84 commits into from
Sep 27, 2023

Commits on Sep 6, 2023

  1. rough refactor

    @AnnaTz
    AnnaTz committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    e28092f View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2023

  1. fixed the call of ivy.multi_head_attention

    @AnnaTz
    AnnaTz committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    3fabf90 View commit details
    Browse the repository at this point in the history

  2. refactored/extended tests

    @AnnaTz
    AnnaTz committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    6286431 View commit details
    Browse the repository at this point in the history

  3. added the new args to the instances

    @AnnaTz
    AnnaTz committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    2dd2478 View commit details
    Browse the repository at this point in the history

  4. corrections in the compositional implementation

    @AnnaTz
    AnnaTz committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    2f88088 View commit details
    Browse the repository at this point in the history

  5. fixed bug in dtype sampling

    @AnnaTz
    AnnaTz committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    4d979ee View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2023

  1. Merge branch 'main' into mha

    @AnnaTz
    AnnaTz committed Sep 8, 2023
    Configuration menu
    Copy the full SHA
    236377b View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2023

  1. Merge branch 'main' into mha

    @AnnaTz
    AnnaTz committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    116c891 View commit details
    Browse the repository at this point in the history

  2. corrected pre_embed_dim vs embed_dim

    @AnnaTz
    AnnaTz committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    c88852d View commit details
    Browse the repository at this point in the history

  3. avoid overflows

    @AnnaTz
    AnnaTz committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    b944063 View commit details
    Browse the repository at this point in the history

  4. fix pre-embedding bug

    @AnnaTz
    AnnaTz committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    8794146 View commit details
    Browse the repository at this point in the history

  5. torch doesn't support self-attention

    @AnnaTz
    AnnaTz committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    b37e5bd View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2023

  1. Merge branch 'main' into mha

    @AnnaTz
    AnnaTz committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    65ba5b7 View commit details
    Browse the repository at this point in the history

  2. avoid overflow

    @AnnaTz
    AnnaTz committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    d637691 View commit details
    Browse the repository at this point in the history

  3. corrected argument order

    @AnnaTz
    AnnaTz committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    a1076d0 View commit details
    Browse the repository at this point in the history

  4. values won't be the same when training due to random dropout

    @AnnaTz
    AnnaTz committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    9d38d95 View commit details
    Browse the repository at this point in the history

  5. updated array ranges in the test

    @AnnaTz
    AnnaTz committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    db6a137 View commit details
    Browse the repository at this point in the history

  6. fixed some bugs of embedding vs pre-embedding dims

    @AnnaTz
    AnnaTz committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    cd63554 View commit details
    Browse the repository at this point in the history

  7. forgot to remove code when moving it

    @AnnaTz
    AnnaTz committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    03acf9d View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2023

  1. wrong dim expanded in torch backend

    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    9192fa0 View commit details
    Browse the repository at this point in the history

  2. shortened the partial mixed handler check

    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    71a4707 View commit details
    Browse the repository at this point in the history

  3. native torch always returns a tuple

    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    9548879 View commit details
    Browse the repository at this point in the history

  4. fixed attention mask bugs

    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    83c3671 View commit details
    Browse the repository at this point in the history

  5. have same_pre_embed_dim affect all cases in the strategy, so that the… 

    … torch backend is properly tested
    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    6cbb35d View commit details
    Browse the repository at this point in the history

  6. fixed shape mismatches in return

    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    e8a36e0 View commit details
    Browse the repository at this point in the history

  7. ivy.where is inconsistent for non bool inputs

    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    7e449c5 View commit details
    Browse the repository at this point in the history

  8. native torch doesn't return the correct weights when causal is applied

    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    495b820 View commit details
    Browse the repository at this point in the history

  9. fixed the return dtypes of test strategy

    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    95c79d0 View commit details
    Browse the repository at this point in the history

  10. native torch can't have None out_proj_weights

    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    8648f1e View commit details
    Browse the repository at this point in the history

  11. fixed key_padding_mask

    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    45f1608 View commit details
    Browse the repository at this point in the history

  12. fixed dimension order in torch backend

    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    629fee1 View commit details
    Browse the repository at this point in the history

  13. fixing bias_k/bias_v

    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    188404c View commit details
    Browse the repository at this point in the history

  14. fixed mask dtypes

    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    ccb93a3 View commit details
    Browse the repository at this point in the history

  15. fixed bias_k/bias_v

    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    c96a5fb View commit details
    Browse the repository at this point in the history

  16. fixed zero attention

    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    e41ded9 View commit details
    Browse the repository at this point in the history

  17. fixed return shape mismatches

    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    6791b4a View commit details
    Browse the repository at this point in the history

  18. fixed static_k/static_v

    @AnnaTz
    AnnaTz committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    b88276d View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2023

  1. Merge branch 'main' into mha

    @AnnaTz
    AnnaTz committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    5a46d04 View commit details
    Browse the repository at this point in the history

  2. reshape expects integers

    @AnnaTz
    AnnaTz committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    4e176c9 View commit details
    Browse the repository at this point in the history

  3. leave a todo for the problematic cases and exclude them for now

    @AnnaTz
    AnnaTz committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    9ee4b00 View commit details
    Browse the repository at this point in the history

  4. fixed typo

    @AnnaTz
    AnnaTz committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    1c5c7a2 View commit details
    Browse the repository at this point in the history

  5. refactored torch helper to be able to re-use it in the frontend

    @AnnaTz
    AnnaTz committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    ed96ca5 View commit details
    Browse the repository at this point in the history

  6. completed the frontend function and test

    @AnnaTz
    AnnaTz committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    4c6508a View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2023

  1. fixed dtype inconsistency

    @AnnaTz
    AnnaTz committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    c7e5279 View commit details
    Browse the repository at this point in the history

  2. removed faulty case from the test strategy

    @AnnaTz
    AnnaTz committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    123fcb6 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into mha

    @AnnaTz
    AnnaTz committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    58aa54d View commit details
    Browse the repository at this point in the history

  4. fixed wrong dimension reference in frontend

    @AnnaTz
    AnnaTz committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    51b0b54 View commit details
    Browse the repository at this point in the history

  5. fixed scale bug in the frontend

    @AnnaTz
    AnnaTz committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    6d9bb23 View commit details
    Browse the repository at this point in the history

  6. fixed key_padding_mask in ivy function

    @AnnaTz
    AnnaTz committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    0953097 View commit details
    Browse the repository at this point in the history

  7. fixed kwarg errors in the torch backend

    @AnnaTz
    AnnaTz committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    a702baf View commit details
    Browse the repository at this point in the history

  8. fixed a couple of left over bugs in the frontend

    @AnnaTz
    AnnaTz committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    ba3af23 View commit details
    Browse the repository at this point in the history

  9. didn't need to batch in the unbatched case in the torch backend

    @AnnaTz
    AnnaTz committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    c9b55e0 View commit details
    Browse the repository at this point in the history

  10. similar fix for the unbatched case in the frontend

    @AnnaTz
    AnnaTz committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    de3346c View commit details
    Browse the repository at this point in the history

  11. fixed list error; list of an array was separating the rows

    @AnnaTz
    AnnaTz committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    c680849 View commit details
    Browse the repository at this point in the history

  12. mask can be either bool or same type as query

    @AnnaTz
    AnnaTz committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    4c25ac4 View commit details
    Browse the repository at this point in the history

  13. fixed wrong dimension reference in the frontend

    @AnnaTz
    AnnaTz committed Sep 19, 2023
    Configuration menu
    Copy the full SHA
    edd2f39 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2023

  1. Merge branch 'main' into mha

    @AnnaTz
    AnnaTz committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    71fc93c View commit details
    Browse the repository at this point in the history

  2. fixed wrong dimension reference in the frontend

    @AnnaTz
    AnnaTz committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    c582622 View commit details
    Browse the repository at this point in the history

  3. force original mask dtypes

    @AnnaTz
    AnnaTz committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    fc31f2e View commit details
    Browse the repository at this point in the history

  4. new detailed docstring

    @AnnaTz
    AnnaTz committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    4b4e8a2 View commit details
    Browse the repository at this point in the history

  5. tolerance

    @AnnaTz
    AnnaTz committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    859bbdd View commit details
    Browse the repository at this point in the history

  6. finally fixed the dimension referencing issue in the static case

    @AnnaTz
    AnnaTz committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    a5ce167 View commit details
    Browse the repository at this point in the history

  7. previous commit continued

    @AnnaTz
    AnnaTz committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    1e9daf2 View commit details
    Browse the repository at this point in the history

  8. fixed bug of wrong dtypes passed to test_frontend_function

    @AnnaTz
    AnnaTz committed Sep 20, 2023
    Configuration menu
    Copy the full SHA
    08d42a6 View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2023

  1. Merge branch 'main' into mha

    @AnnaTz
    AnnaTz committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    acb5abb View commit details
    Browse the repository at this point in the history

  2. fixed attention mask, numeric masks are added to the weights

    @AnnaTz
    AnnaTz committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    baf0e21 View commit details
    Browse the repository at this point in the history

  3. emb_dim should be integer

    @AnnaTz
    AnnaTz committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    7952491 View commit details
    Browse the repository at this point in the history

  4. refactored the shaping of the matrix multiplications

    @AnnaTz
    AnnaTz committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    2f00897 View commit details
    Browse the repository at this point in the history

  5. similar attention mask fix for torch backend

    @AnnaTz
    AnnaTz committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    bf56f48 View commit details
    Browse the repository at this point in the history

  6. make scale positive

    @AnnaTz
    AnnaTz committed Sep 21, 2023
    Configuration menu
    Copy the full SHA
    9308c66 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2023

  1. mask refactor

    @AnnaTz
    AnnaTz committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    a2d2a28 View commit details
    Browse the repository at this point in the history

  2. key_padding_mask needs to be converted to float like attention_mask

    @AnnaTz
    AnnaTz committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    7907596 View commit details
    Browse the repository at this point in the history

  3. corrected case of is_causal and need_weights in the frontend

    @AnnaTz
    AnnaTz committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    c2e594f View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2023

  1. fixed is_causal + more masking bugs

    @AnnaTz
    AnnaTz committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    827086e View commit details
    Browse the repository at this point in the history

  2. 🤖 Lint code

    @ivy-branch
    ivy-branch committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    fda3ee1 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2023

  1. added argument batch_first

    @AnnaTz
    AnnaTz committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    dca30d6 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into mha

    @AnnaTz
    AnnaTz committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    b350187 View commit details
    Browse the repository at this point in the history

  3. 🤖 Lint code

    @ivy-branch
    ivy-branch committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    72380d1 View commit details
    Browse the repository at this point in the history

  4. test(backend handler): added tests for inplace update warnings. (#26067)

    @Madjid-CH @AnnaTz
    Madjid-CH authored and AnnaTz committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    803e2a4 View commit details
    Browse the repository at this point in the history

  5. fix(ivy): Added missing wrapper import

    @AnnaTz
    AnnaTz committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    8da76fe View commit details
    Browse the repository at this point in the history

  6. minor changes

    @AnnaTz
    AnnaTz committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    c2118d6 View commit details
    Browse the repository at this point in the history

  7. refactored & fixed static_k/static_v

    @AnnaTz
    AnnaTz committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    47be090 View commit details
    Browse the repository at this point in the history

  8. renamed variable

    @AnnaTz
    AnnaTz committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    8f5ee33 View commit details
    Browse the repository at this point in the history

  9. fixed bug in testing strategy

    @AnnaTz
    AnnaTz committed Sep 27, 2023
    Configuration menu
    Copy the full SHA
    c99bdbd View commit details
    Browse the repository at this point in the history