Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for FP16/BF16 in train_gpt2.cu (1.86x Perf) #218

Merged
merged 12 commits into from
Apr 23, 2024
Merged

Support for FP16/BF16 in train_gpt2.cu (1.86x Perf) #218

merged 12 commits into from
Apr 23, 2024

Commits on Apr 22, 2024

  1. WIP support for FP16/BF16 in train_gpt2.cu (compiles, not correct res… 

    …ults)
    @ademeure
    ademeure committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    f35adbe View commit details
    Browse the repository at this point in the history

  2. Gradients now working in BF16 mode! (still need adam etc...)

    @ademeure
    ademeure committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    58df262 View commit details
    Browse the repository at this point in the history

  3. Fully working BF16 training! (very hacky Adam, and converges much slo… 

    …wer than FP32 for now)
    @ademeure
    ademeure committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    d605b99 View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2024

  1. Updated FP16/BF16 mix-and-match ncluding stochastic rounding

    @ademeure
    ademeure committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    85290a5 View commit details
    Browse the repository at this point in the history

  2. BF16/FP16 Attention + bug fixes

    @ademeure
    ademeure committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    bdc661a View commit details
    Browse the repository at this point in the history

  3. Good news: it works. Bad news: 16-bit atomics are *incredibly slow* :… 

    …( (is it doing CAS?)
    @ademeure
    ademeure committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    7c193bd View commit details
    Browse the repository at this point in the history

  4. Added floatN to choose FP32 vs FP16 for layernorm (parameters/gradien… 

    …ts only)
    @ademeure
    ademeure committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    8775856 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'karpathy/master' into linear16

    @ademeure
    ademeure committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    c1992a1 View commit details
    Browse the repository at this point in the history

  6. Fixes for merge with latest, now ~86% faster!

    @ademeure
    ademeure committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    53dc40e View commit details
    Browse the repository at this point in the history

  7. Fill in param_sizeof in a slightly less terrible way

    @ademeure
    ademeure committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    a876485 View commit details
    Browse the repository at this point in the history

  8. fix typo

    @ademeure
    ademeure committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    91ec92f View commit details
    Browse the repository at this point in the history

  9. tiny fixes to make fp8 work in the future (missing conversions)

    @ademeure
    ademeure committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    0333e36 View commit details
    Browse the repository at this point in the history