Skip to content

huggingface/candle-layer-norm

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

14 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Candle Cuda Layer Norm

Layer Norm fused operation for the Candle ML framework.

This Layer was adapted from https://github.com/Dao-AILab/flash-attention/tree/main/csrc/layer_norm.

It implements fused dropout + residual + LayerNorm, building on Apex's FastLayerNorm.

Major changes:

  • Add residual.
  • Make it work for both pre-norm and post-norm architecture.
  • Support more hidden dimensions (all dimensions divisible by 8, up to 8192).
  • Implement RMSNorm as an option.

About

No description, website, or topics provided.

Resources

License

Apache-2.0, MIT licenses found

Licenses found

Apache-2.0
LICENSE-APACHE
MIT
LICENSE-MIT

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published