Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RoPE loses precision for Llama / Gemma + Gemma logits.float() #29285

Merged
merged 20 commits into from
Feb 28, 2024
Merged

RoPE loses precision for Llama / Gemma + Gemma logits.float() #29285

merged 20 commits into from
Feb 28, 2024
+23 −8

Commits on Feb 26, 2024

  1. Update modeling_llama.py 

    Llama - Force float32 since bfloat16 loses precision on long contexts
    @danielhanchen
    danielhanchen committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    7a25720 View commit details
    Browse the repository at this point in the history

  2. Update modeling_llama.py

    @danielhanchen
    danielhanchen committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    db8237f View commit details
    Browse the repository at this point in the history

  3. Update modeling_gemma.py 

    Fix RoPE and logits.float()
    @danielhanchen
    danielhanchen committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    3de95c4 View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2024

  1. Merge branch 'huggingface:main' into main

    @danielhanchen
    danielhanchen committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    9e5cbb0 View commit details
    Browse the repository at this point in the history

  2. @torch.no_grad()

    @danielhanchen
    danielhanchen committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    99d564e View commit details
    Browse the repository at this point in the history

  3. @torch.no_grad()

    @danielhanchen
    danielhanchen committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    d0c08bf View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2024

  1. Merge branch 'huggingface:main' into main

    @danielhanchen
    danielhanchen committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    bd3a214 View commit details
    Browse the repository at this point in the history

  2. Cos, Sin to float32

    @danielhanchen
    danielhanchen committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    abffebb View commit details
    Browse the repository at this point in the history

  3. cos, sin to float32

    @danielhanchen
    danielhanchen committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    c2e31bf View commit details
    Browse the repository at this point in the history

  4. Update src/transformers/models/gemma/modeling_gemma.py 

    Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
    @danielhanchen @ArthurZucker
    danielhanchen and ArthurZucker committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    f487800 View commit details
    Browse the repository at this point in the history

  5. Update src/transformers/models/llama/modeling_llama.py 

    Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
    @danielhanchen @ArthurZucker
    danielhanchen and ArthurZucker committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    c852675 View commit details
    Browse the repository at this point in the history

  6. Resolve PR conflicts

    @danielhanchen
    danielhanchen committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    1a50a4b View commit details
    Browse the repository at this point in the history

  7. Fix RoPE for llama

    @danielhanchen
    danielhanchen committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    b860a22 View commit details
    Browse the repository at this point in the history

  8. Revert "Fix RoPE for llama" 

    This reverts commit b860a22.
    @danielhanchen
    danielhanchen committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    790e4a3 View commit details
    Browse the repository at this point in the history

  9. Merge remote-tracking branch 'upstream/main'

    @danielhanchen
    danielhanchen committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    06c7634 View commit details
    Browse the repository at this point in the history

  10. Fix RoPE for llama

    @danielhanchen
    danielhanchen committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    aa03a43 View commit details
    Browse the repository at this point in the history

  11. RoPE device

    @danielhanchen
    danielhanchen committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    5730a50 View commit details
    Browse the repository at this point in the history

  12. Autocast device type

    @danielhanchen
    danielhanchen committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    31cea3b View commit details
    Browse the repository at this point in the history

  13. RoPE

    @danielhanchen
    danielhanchen committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    ae9957f View commit details
    Browse the repository at this point in the history

  14. RoPE isinstance

    @danielhanchen
    danielhanchen committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    ec9ef17 View commit details
    Browse the repository at this point in the history