Skip to content

v0.2.8

Choose a tag to compare

View all tags
@Disty0 Disty0 released this 10 Oct 17:55
· 41 commits to main since this release
v0.2.8
8aa3dac

New Features

  • Add double-quantization support for scales and zero points with scale_dtype, zero_point_dtype and use_codebook_scale.
    • The default zero_point_dtype value None means it will use the same value from scale_dtype.
    • NVFP4 is an example of double-quantization using:
      • weights_dtype = float4_e2m1fn
      • scale_dtype = float8_e4m3fn
      • group_size = 16

Changes

  • Add default values to SDNQDequantizer.
  • Refactor SDNQTensor to use sdnq.common.sdnq_keys instead of hard-coded keys.
  • Move the SDNQ Atten backward pass dQ and dK sm_scale scaling to occur after the matmul instead of scaling dS before the matmul (aka the 3rd fix from KohakuFA).

Fixes

  • Fix kahan_buffer init on optimizers.
  • Fix float attn_mask accuracy on SDNQ Atten.

Full Changelog: v0.2.7...v0.2.8

Assets 2
Loading