Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cuDNN Forward Attention + FP16 non-cuDNN version in /dev/cuda/ #215

Closed
wants to merge 8 commits into from
Closed

cuDNN Forward Attention + FP16 non-cuDNN version in /dev/cuda/ #215

wants to merge 8 commits into from

Commits on Apr 21, 2024

  1. Initial working FP16 cuDNN Forward Attention (+ FP16 port of kernel 4)

    @ademeure
    ademeure committed Apr 21, 2024
    Configuration menu
    Copy the full SHA
    da987fa View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'karpathy/master' into forwardcudnn

    @ademeure
    ademeure committed Apr 21, 2024
    Configuration menu
    Copy the full SHA
    0109bd1 View commit details
    Browse the repository at this point in the history

  3. WIP - cuBLAS Ex() not working as expected...

    @ademeure
    ademeure committed Apr 21, 2024
    Configuration menu
    Copy the full SHA
    8a93ecf View commit details
    Browse the repository at this point in the history

  4. Merge branch 'forwardcudnn' of https://github.com/ademeure/llm.c into… 

    … forwardcudnn
    @ademeure
    ademeure committed Apr 21, 2024
    Configuration menu
    Copy the full SHA
    60c258a View commit details
    Browse the repository at this point in the history

  5. Fix FP16/BF16 kernel version issues

    @ademeure
    ademeure committed Apr 21, 2024
    Configuration menu
    Copy the full SHA
    6ed6e99 View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. Use -DENABLE_CUDNN for cuDNN path instead, and enable TF32 for attent… 

    …ion_forward.cu by default
    @ademeure
    ademeure committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    929ad2f View commit details
    Browse the repository at this point in the history

  2. extra comments + tiny fix

    @ademeure
    ademeure committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    33a31a7 View commit details
    Browse the repository at this point in the history

  3. remove unintentional train_gpu2.cu change

    @ademeure
    ademeure committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    2c47bd4 View commit details
    Browse the repository at this point in the history