Quantization Params Explanation + llama.cpp Relation

[yukiarimo]

Hello! I saw there several parameters for quantizing model in MLX:

    parser.add_argument(
        "-q", "--quantize", help="Generate a quantized model.", action="store_true"
    )
    parser.add_argument(
        "--q-group-size", help="Group size for quantization.", type=int, default=64
    )
    parser.add_argument(
        "--q-bits", help="Bits per weight for quantization.", type=int, default=4
    )
    parser.add_argument(
        "--dtype",
        help="Type to save the parameters, ignored if -q is given.",
        type=str,
        choices=["float16", "bfloat16", "float32"],
        default="float16",
    )

The models that I use are between 7-14 B parameters. Can you please explain what’s that group size and bits per weight and which one should I set to what?

Cause it llama.cpp I had GGUF and there it was like my favorites: Q3_K_L, Q4_K_M, Q5_K_M, Q6_K. So having a table (if it’s even possible to compare) would be nice!

Thanks!

[Blaizzy]

group size default of 64 works well for most models

q-bits gives you the precision (3, 4, 6 or 8bit) -> equivalent to Q3, Q4, Q6, Q8

Dtype sets the data type

[Blaizzy]

I would appreciate if you could make the table using all the models existent on HF mlx-community repo.

[yukiarimo]

Thanks!

1. So, what’s the group size for anyway? And how much it affects the performance, cause in llama.cpp there's no group size!
2. What is better (and faster) for inference here, bf16 or fp16?

[Blaizzy]

group size is how many values to group together when quantizing.

[yukiarimo]

Thanks!