New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GaLore Finetuning Example #10722
GaLore Finetuning Example #10722
Conversation
|
||
**--repo-id-or-model-path** : default to `meta-llama/Llama-2-7b-hf`, and you can also specify your local model path. | ||
|
||
**--data-path** : default to `g-ronimo/oasst2_top4k_en`, and you can also specify your local datal path. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please check if this dataset is ok to use.
https://github.com/analytics-zoo/nano/issues/1112#issuecomment-1978363264
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
change dataset to HuggingFaceH4/helpful_instructions, a chat completion dataset, which is on our white list
parser.add_argument('--repo-id-or-model-path', type=str, default="meta-llama/Llama-2-7b-chat-hf", | ||
help='The huggingface repo id for the Llama2 (e.g. `meta-llama/Llama-2-7b-chat-hf` and `meta-llama/Llama-2-13b-chat-hf`) to be downloaded' | ||
', or the path to the huggingface checkpoint folder') | ||
parser.add_argument('--data-path', type=str, default="g-ronimo/oasst2_top4k_en", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
dataset need to be verified
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Description
add an example of new fine-tuning algorithm on IPEX-LLM llama2 7b
#nano
1. Why the change?
as above
2. User API changes
How to Run
3. Summary of the change
GaLore Finetuning Example
4. How to test?
5. New dependencies
galore-torch
trl==0.8.1