[Feature Request] Fp6 datatype support

[TedThemistokleous]

Describe the feature request

Allow for Onnxruntime to parse and support the FP6 datatype for Onnx models.

Currently in the Onnx spec this doesn't exist yet but I've gotten a feature request started on ONNX as well

onnx/onnx#7048

Lookng to leverage this within MIGraphX as well as the MIGraphX Execution Provider for prequantized models

Describe scenario use case

Having execution providers take in fp6 (any bit scheme) quantized models and support inference

[TedThemistokleous]

cc @tianleiwu Please let me know what's required to add support. I see you have tensorproto datatypes in the repo at a quick glance but if there's any other files as well require please let me know. Will be looking to the coming months to add this

[tianleiwu]

@TedThemistokleous,

For LLM, typical onnx usage is mixing 4bits and 8bits. Most weights can be quantized to 4 bits, some layers need more bits and we normally use 8 bits (like lm_heads).

Any experiment results on mixed precision of W4A8+W8A8 vs W4A8+W6A8 (like memory usage reduction rate, accuracy loss, throughput gain/loss) on some popular LLM model?

[TedThemistokleous]

Not yet but we'd like to support that data type and have the option for customers/our team to experiment when trying to find the optimial setup. Fp6 is a middle ground between fp4 and fp8 and depending on the Exponent/mantissa used vary the precision.

My understanding is there's two fp6 datatypes E3M2 and E2M3 to represent values

[tianleiwu]

We can add the support of FP6 once onnx adds it.

Feature	NVIDIA CUDA (12.9+)	AMD ROCm (7.0+)
FP6 Formats	e2m3, e3m2	e2m3, e3m2
GPU Support	Hopper, Blackwell (Compute Capability 10.0a+)	Instinct MI350X, MI355X (CDNA 4)

Related Doc:
https://docs.nvidia.com/cuda/cuda-math-api/cuda_math_api/group__CUDA__MATH__FP6__MISC.html
https://rocm.docs.amd.com/projects/HIP/en/docs-develop/reference/low_fp_types.html

[TedThemistokleous]

Also useful https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf