How to quantization INT8 / FP16

[Jeremy-J-J]

How to quantization INT8 / FP16?

Define_string_opt("--quantization", g_flag_quantization, "", "declare **json file** saved quantization information");

what does the json file look like？

[Si-XU]

Hi, OpenPPL for cuda only supports fp16 right now. We will support int8 ASAP.
When you input a fp32 data file, OpenPPL will automatically transfer to fp16.
"--quantization" is designed for int8, so you don't need to use it.