Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement DoRA #1474

Merged
merged 21 commits into from Feb 27, 2024
Merged

Implement DoRA #1474

merged 21 commits into from Feb 27, 2024
+352 −30

Commits on Feb 16, 2024

  1. [WIP] Try to implement DoRA 

    https://arxiv.org/abs/2402.09353
    @BenjaminBossan
    BenjaminBossan committed Feb 16, 2024
    Configuration menu
    Copy the full SHA
    6242f4a View commit details
    Browse the repository at this point in the history

Commits on Feb 19, 2024

  1. Merge branch 'main' into feat-dora

    @BenjaminBossan
    BenjaminBossan committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    32ffc50 View commit details
    Browse the repository at this point in the history

  2. Better var name

    @BenjaminBossan
    BenjaminBossan committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    53a5e25 View commit details
    Browse the repository at this point in the history

  3. Update all signatures of update_layer 

    Might make sense to refactor this in the future to avoid a signficant
amount of code duplication.
    @BenjaminBossan
    BenjaminBossan committed Feb 19, 2024
    Configuration menu
    Copy the full SHA
    18fb476 View commit details
    Browse the repository at this point in the history

Commits on Feb 20, 2024

  1. Fix to how weight_norm is calculated

    @BenjaminBossan
    BenjaminBossan committed Feb 20, 2024
    Configuration menu
    Copy the full SHA
    e4a677f View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2024

  1. Fix implementation

    @BenjaminBossan
    BenjaminBossan committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    c42bb0a View commit details
    Browse the repository at this point in the history

  2. Fix a few failing tests

    @BenjaminBossan
    BenjaminBossan committed Feb 21, 2024
    Configuration menu
    Copy the full SHA
    b915ea2 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2024

  1. Some adjustments to correctly consider scaling

    @BenjaminBossan
    BenjaminBossan committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    8bd22d7 View commit details
    Browse the repository at this point in the history

  2. Make style

    @BenjaminBossan
    BenjaminBossan committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    ecf7160 View commit details
    Browse the repository at this point in the history

  3. Small fix of calculation of weight norm

    @BenjaminBossan
    BenjaminBossan committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    07f3e43 View commit details
    Browse the repository at this point in the history

  4. Make style

    @BenjaminBossan
    BenjaminBossan committed Feb 22, 2024
    Configuration menu
    Copy the full SHA
    5b22170 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2024

  1. Add documentation for DoRA

    @BenjaminBossan
    BenjaminBossan committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    f15dbae View commit details
    Browse the repository at this point in the history

  2. Improve DoRA init test

    @BenjaminBossan
    BenjaminBossan committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    1e6d1d7 View commit details
    Browse the repository at this point in the history

  3. Address reviewer feedback

    @BenjaminBossan
    BenjaminBossan committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    69a81a6 View commit details
    Browse the repository at this point in the history

  4. Fix for DeepSpeed Zero3 

    Context manager taken from huggingface#1267
    @BenjaminBossan
    BenjaminBossan committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    4a90843 View commit details
    Browse the repository at this point in the history

  5. Fix context call

    @BenjaminBossan
    BenjaminBossan committed Feb 23, 2024
    Configuration menu
    Copy the full SHA
    951ae67 View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2024

  1. Address reviewer feedback

    @BenjaminBossan
    BenjaminBossan committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    9c4edc1 View commit details
    Browse the repository at this point in the history

  2. More efficient forward pass

    @BenjaminBossan
    BenjaminBossan committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    4bf7346 View commit details
    Browse the repository at this point in the history

  3. Make style

    @BenjaminBossan
    BenjaminBossan committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    ebdda07 View commit details
    Browse the repository at this point in the history

  4. Update comment

    @BenjaminBossan
    BenjaminBossan committed Feb 26, 2024
    Configuration menu
    Copy the full SHA
    6194726 View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2024

  1. Make deepspeed check work with older transformers

    @BenjaminBossan
    BenjaminBossan committed Feb 27, 2024
    Configuration menu
    Copy the full SHA
    51d4919 View commit details
    Browse the repository at this point in the history