Allow user to use custom calibration data for quantization #27
Conversation
|
Thanks for this PR, TheBloke also asked for this. Will review it later. Before this is merged, I would also like to create two examples of how to use the functionality with either a string pointing to a huggingface dataset or a list of preprocessed data. |
awq/utils/calib_data.py
Outdated
| if data == "pileval": | ||
| dataset = load_dataset("mit-han-lab/pile-val-backup", split="validation") | ||
| else: | ||
| raise NotImplementedError |
There was a problem hiding this comment.
Should work. Might want to find a way define the split instead of defaulting to train, though.
| raise NotImplementedError | |
| dataset = load_dataset(data, split="train") |
There was a problem hiding this comment.
The defaulting to train might be solved by adding a kwarg with that defaults to validation which could be used in L9 and L11.
There was a problem hiding this comment.
I agree that we need to not raise an exception here. Instead, we should try to load the dataset by the actual string that was passed and load the split by another variable passed. We could default to the validation split as this should be a small enough dataset for calibration yet scientifically sound enough since we would use the test split to measure perplexity.
|
What's the benefit of using custom data? |
Hi,
If you have some time to review these changes,
It should allow to use custom dataset (
List[str]) for calibration part.Thx :)