Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add qwen2 #28436

Merged
merged 76 commits into from
Jan 17, 2024
Merged

Add qwen2 #28436

merged 76 commits into from
Jan 17, 2024

Commits on Jan 10, 2024

  1. add config, modeling, and tokenization

    @JustinLin610
    JustinLin610 committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    f5bf018 View commit details
    Browse the repository at this point in the history

  2. add auto and init

    @JustinLin610
    JustinLin610 committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    3226f3a View commit details
    Browse the repository at this point in the history

  3. update readme

    @JustinLin610
    JustinLin610 committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    52ab139 View commit details
    Browse the repository at this point in the history

  4. update readme

    @JustinLin610
    JustinLin610 committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    0fe20cd View commit details
    Browse the repository at this point in the history

  5. update team name

    @JustinLin610
    JustinLin610 committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    8299266 View commit details
    Browse the repository at this point in the history

  6. fixup

    @JustinLin610
    JustinLin610 committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    b8a8e22 View commit details
    Browse the repository at this point in the history

  7. fixup

    @JustinLin610
    JustinLin610 committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    55802f3 View commit details
    Browse the repository at this point in the history

  8. update config

    @JustinLin610
    JustinLin610 committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    6b587b8 View commit details
    Browse the repository at this point in the history

  9. update code style

    @JustinLin610
    JustinLin610 committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    dfe7d2e View commit details
    Browse the repository at this point in the history

  10. update for fixup

    @JustinLin610
    JustinLin610 committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    525de40 View commit details
    Browse the repository at this point in the history

  11. update for fixup

    @JustinLin610
    JustinLin610 committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    7f6f5e2 View commit details
    Browse the repository at this point in the history

  12. update for fixup

    @JustinLin610
    JustinLin610 committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    36fcf65 View commit details
    Browse the repository at this point in the history

  13. update for testing

    @JustinLin610
    JustinLin610 committed Jan 10, 2024
    Configuration menu
    Copy the full SHA
    42f561a View commit details
    Browse the repository at this point in the history

Commits on Jan 11, 2024

  1. update for testing

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    105d98f View commit details
    Browse the repository at this point in the history

  2. fix bug for config and tokenization

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    47e6e9e View commit details
    Browse the repository at this point in the history

  3. fix bug for bos token

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    5061a26 View commit details
    Browse the repository at this point in the history

  4. not doctest

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    0987dab View commit details
    Browse the repository at this point in the history

  5. debug tokenizer

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    bf4e928 View commit details
    Browse the repository at this point in the history

  6. not doctest

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    2658baf View commit details
    Browse the repository at this point in the history

  7. debug tokenization

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    69b03e7 View commit details
    Browse the repository at this point in the history

  8. debug init for tokenizer

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    725ab32 View commit details
    Browse the repository at this point in the history

  9. fix style

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    57264f3 View commit details
    Browse the repository at this point in the history

  10. update init

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    0489d0f View commit details
    Browse the repository at this point in the history

  11. delete if in token auto

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    e3b60a6 View commit details
    Browse the repository at this point in the history

  12. add tokenizer doc

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    e1b4a8c View commit details
    Browse the repository at this point in the history

  13. add tokenizer in init

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    22d3402 View commit details
    Browse the repository at this point in the history

  14. Update dummy_tokenizers_objects.py

    @jklj077
    jklj077 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    c05a7be View commit details
    Browse the repository at this point in the history

  15. Merge pull request #1 from jklj077/patch-1 

    Update dummy_tokenizers_objects.py
    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    783eec8 View commit details
    Browse the repository at this point in the history

  16. update

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    c24ee57 View commit details
    Browse the repository at this point in the history

  17. update

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    0f83bc1 View commit details
    Browse the repository at this point in the history

  18. debug

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    e68a65c View commit details
    Browse the repository at this point in the history

  19. Merge branch 'huggingface:main' into add_qwen2

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    03ff228 View commit details
    Browse the repository at this point in the history

  20. Update tokenization_qwen2.py

    @jklj077
    jklj077 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    2e00431 View commit details
    Browse the repository at this point in the history

  21. Merge pull request #3 from jklj077/patch-1 

    Update tokenization_qwen2.py
    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    73695ff View commit details
    Browse the repository at this point in the history

  22. debug

    @JustinLin610
    JustinLin610 committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    e041c79 View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2024

  1. Merge branch 'huggingface:main' into add_qwen2

    @JustinLin610
    JustinLin610 committed Jan 12, 2024
    Configuration menu
    Copy the full SHA
    e96c5c2 View commit details
    Browse the repository at this point in the history

Commits on Jan 13, 2024

  1. Update convert_slow_tokenizer.py

    @jklj077
    jklj077 committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    9cda260 View commit details
    Browse the repository at this point in the history

  2. add copies

    renxuancheng.rxc committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    a81deed View commit details
    Browse the repository at this point in the history

  3. Merge pull request #4 from jklj077/patch-3 

    update qwen2 tokenization
    @JustinLin610
    JustinLin610 committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    e652aa9 View commit details
    Browse the repository at this point in the history

  4. add copied from and make style

    @JustinLin610
    JustinLin610 committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    985fe0f View commit details
    Browse the repository at this point in the history

  5. update files map

    renxuancheng.rxc committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    f7beb77 View commit details
    Browse the repository at this point in the history

  6. update test

    renxuancheng.rxc committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    35a7480 View commit details
    Browse the repository at this point in the history

  7. Merge pull request #5 from jklj077/patch-4 

    Update files map so that tests can find he pretrained tokenizers
    @JustinLin610
    JustinLin610 committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    26a24db View commit details
    Browse the repository at this point in the history

  8. fix style

    @JustinLin610
    JustinLin610 committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    f419098 View commit details
    Browse the repository at this point in the history

  9. fix merge reading and update tests

    renxuancheng.rxc committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    6b3247b View commit details
    Browse the repository at this point in the history

  10. Merge pull request #6 from jklj077/patch-6 

    fix merge reading and update tests
    @JustinLin610
    JustinLin610 committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    68968f5 View commit details
    Browse the repository at this point in the history

  11. fix tests

    renxuancheng.rxc committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    9c45e77 View commit details
    Browse the repository at this point in the history

  12. fix tests

    renxuancheng.rxc committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    93bae64 View commit details
    Browse the repository at this point in the history

  13. Merge pull request #7 from jklj077/patch-6 

    fix 3 tests
    @JustinLin610
    JustinLin610 committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    1345aae View commit details
    Browse the repository at this point in the history

  14. fix style

    @JustinLin610
    JustinLin610 committed Jan 13, 2024
    Configuration menu
    Copy the full SHA
    bd57c63 View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2024

  1. debug a variable in readme

    @JustinLin610
    JustinLin610 committed Jan 14, 2024
    Configuration menu
    Copy the full SHA
    25d43f0 View commit details
    Browse the repository at this point in the history

Commits on Jan 15, 2024

  1. Update src/transformers/models/qwen2/configuration_qwen2.py 

    Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
    @JustinLin610 @ArthurZucker
    JustinLin610 and ArthurZucker committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    e68a1b8 View commit details
    Browse the repository at this point in the history

  2. update test and copied from

    @JustinLin610
    JustinLin610 committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    4135193 View commit details
    Browse the repository at this point in the history

  3. fix style

    @JustinLin610
    JustinLin610 committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    8708e50 View commit details
    Browse the repository at this point in the history

  4. update qwen2 tokenization and tests

    renxuancheng.rxc committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    db82439 View commit details
    Browse the repository at this point in the history

  5. Update tokenization_qwen2.py

    @JustinLin610
    JustinLin610 committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    e0c7750 View commit details
    Browse the repository at this point in the history

  6. Merge pull request #8 from jklj077/patch-7 

    update qwen2 tokenization  and tests
    @JustinLin610
    JustinLin610 committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    7c13f74 View commit details
    Browse the repository at this point in the history

  7. delete the copied from after property

    @JustinLin610
    JustinLin610 committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    8ebacad View commit details
    Browse the repository at this point in the history

  8. fix style

    @JustinLin610
    JustinLin610 committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    7b670a8 View commit details
    Browse the repository at this point in the history

  9. update tests

    renxuancheng.rxc committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    304f0dc View commit details
    Browse the repository at this point in the history

  10. Merge branch 'patch-7' of https://github.com/jklj077/transformers int… 

    …o patch-7
    renxuancheng.rxc committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    8b144e1 View commit details
    Browse the repository at this point in the history

  11. Merge pull request #9 from jklj077/patch-7 

    update tests
    @JustinLin610
    JustinLin610 committed Jan 15, 2024
    Configuration menu
    Copy the full SHA
    ca4a8c5 View commit details
    Browse the repository at this point in the history

Commits on Jan 16, 2024

  1. update tests

    renxuancheng.rxc committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    1ce4e4d View commit details
    Browse the repository at this point in the history

  2. Merge pull request #10 from jklj077/patch-7 

    Update qwen2 tokenization tests
    @JustinLin610
    JustinLin610 committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    04c4a75 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'huggingface:main' into add_qwen2

    @JustinLin610
    JustinLin610 committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    93cbb5d View commit details
    Browse the repository at this point in the history

  4. add copied from

    @JustinLin610
    JustinLin610 committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    1e0b937 View commit details
    Browse the repository at this point in the history

  5. fix bugs

    @JustinLin610
    JustinLin610 committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    22f7b1e View commit details
    Browse the repository at this point in the history

  6. update doc

    @JustinLin610
    JustinLin610 committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    69d3f89 View commit details
    Browse the repository at this point in the history

Commits on Jan 17, 2024

  1. add warning for sliding window attention

    renxuancheng.rxc committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    2840f0a View commit details
    Browse the repository at this point in the history

  2. update qwen2 tokenization

    renxuancheng.rxc committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    87f6cf7 View commit details
    Browse the repository at this point in the history

  3. Merge pull request #11 from jklj077/patch-8 

    addressed review comments
    @JustinLin610
    JustinLin610 committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    0a47113 View commit details
    Browse the repository at this point in the history

  4. fix style

    @JustinLin610
    JustinLin610 committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    047e372 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'huggingface:main' into add_qwen2

    @JustinLin610
    JustinLin610 committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    6b6dae0 View commit details
    Browse the repository at this point in the history

  6. Update src/transformers/models/qwen2/modeling_qwen2.py 

    Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
    @JustinLin610 @ArthurZucker
    JustinLin610 and ArthurZucker committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    a4e80ee View commit details
    Browse the repository at this point in the history

  7. Merge branch 'huggingface:main' into add_qwen2

    @JustinLin610
    JustinLin610 committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    39cc60b View commit details
    Browse the repository at this point in the history

  8. fix tokenizer fast

    @JustinLin610
    JustinLin610 committed Jan 17, 2024
    Configuration menu
    Copy the full SHA
    a8b0c3c View commit details
    Browse the repository at this point in the history