Support prequant qwen3 #10839
Support prequant qwen3 #10839
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10839
Note: Links to docs will display an error until the docs builds have been completed. ❗ 1 Active SEVsThere are 1 currently active SEVs. If your PR is affected, please view them below: ❌ 3 New FailuresAs of commit 880c99b with merge base d7201ab ( NEW FAILURES - The following jobs have failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
This PR needs a
|
| converted_state_dict["output.weight"] = converted_state_dict[ | ||
| "tok_embeddings.weight" | ||
| ] | ||
| # If lm_head.weight is not present, assume tied embeddings (e.g., 0.6b and 4b models) |
There was a problem hiding this comment.
| # If lm_head.weight is not present, assume tied embeddings (e.g., 0.6b and 4b models) | |
| # If lm_head.weight is not present, assume tied embeddings (0.6b, 1.7b, and 4b models) |
| "tok_embeddings.weight" | ||
| ] | ||
| # If lm_head.weight is not present, assume tied embeddings (e.g., 0.6b and 4b models) | ||
| if "lm_head.weight" not in state_dict: |
There was a problem hiding this comment.
lm_head is present in the hf checkpoints even if they are tied embeddings, it will just be the same weights as the tok_embeddings
There was a problem hiding this comment.
Hmmm, are you sure it's there? I thought when config. tie_word_embeddings = true, it might not be there, but gets materialized during a tie_weights() command on the HF model.
In any case, if it is there, it's covered by the regular loop through keys and this logic is not executed. If it's not there, this sets lm_head's weight to the embeddings.
There was a problem hiding this comment.
Yeah it's here, https://huggingface.co/Qwen/Qwen3-0.6B/tree/main?show_file_info=model.safetensors. Also I remember seeing it while debugging the checkpoint. But sure, in that case could you reword the comment?
There was a problem hiding this comment.
Reworded the comment a little.
I'm not convinced by https://huggingface.co/Qwen/Qwen3-0.6B/tree/main?show_file_info=model.safetensors because it's just metadata, and doesn't prove anything about what is stored in the file.
It does not look like lm_head is present in the safetensors when I unpack them locally. Perhaps you looked at the checkpoint after running your script? (Which copied the embedding tensors into lm_head).
There was a problem hiding this comment.
Ok, if you double checked then that's fine!
| ) | ||
| parser.add_argument( | ||
| "input_dir", | ||
| "input_dir_or_checkpoint", |
There was a problem hiding this comment.
not a big fan of specifying something that could be either a dir or a file, if pytorch_model.bin is the filename for the quantized checkpoint going forward i'd rather just specify the checkpoint and search for that
There was a problem hiding this comment.
how do you usually download the directories from HF?
There was a problem hiding this comment.
e.g. huggingface-cli download ibm-granite/granite-3b-code-instruct-128k gets you the entire directory
There was a problem hiding this comment.
I changed to use directories.
I can test if there is no CI for it. |
|
Yeah if you test it and it works, feel free to merge pending comments |
Checked convert script works with original Qwen3 on HF |
As titled