-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fine-tuning #14
Comments
Hi Nooshin, |
Hi Mononita,
Thank you for your reply. I have an ECG dataset and I want to do AFib
classification using MOMENT. Actually, now I used the current pre-trained
model to extract embeddings. However, the result was not good. I think
running a fine-tuning step can increase the performance.
Best,
Nooshin
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Virus-free.www.avg.com
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
…On Wed, May 22, 2024 at 1:56 PM Mononito Goswami ***@***.***> wrote:
Hi Nooshin,
Thanks for your interest in MOMENT! Could you please tell us a bit more
about the tasks you want to use MOMENT for?
—
Reply to this email directly, view it on GitHub
<#14 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/A3EZUS7HANBPDFEUDCS5JLTZDTL5BAVCNFSM6AAAAABIDVI5DWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRVGQZDQOJWGM>
.
You are receiving this because you authored the thread.Message ID:
***@***.***
com>
|
Hi Nooshin, Some possible reasons for lack of performance may be:
Recommendations for Fine-tuningI would recommend fine-tuning the model using masked reconstruction. from momentfm import MOMENTPipeline
model = MOMENTPipeline.from_pretrained(
"AutonLab/MOMENT-1-large",
model_kwargs={
'task_name': 'reconstruction',
'forecast_horizon': 192,
'head_dropout': 0.1,
'weight_decay': 0,
'freeze_encoder': False, # Freeze the patch embedding layer. True by default.
'freeze_embedder': False, # Freeze the transformer encoder. True by default.
'freeze_head': False, # False by default
},
) By default the encoder and the patch embedding layer are frozen. I would recommend not freezing them for your experiment. I recommend you check the forecasting script on more details on how to fine-tune the model. Fine-tuning MOMENT is just like fine-tuning any other deep learning model! I would also suggest looking at parameter-efficient fine-tuning techniques such as LoRA for fine-tuning. LoRa might make the fine-tuning process more efficient and may also improve predictive accuracy. Questions
Hope this helps! Let us know if you have any more questions! |
Thank you for your explanation! Actually, I am looking for the research code of MOMENT so that I can modify the fine-tuning process by freezing different layers and adding some task-relevant adapter layers. I noticed that even when we load the model in classification mode, it initially loads in reconstruction mode. My experiment was on an imbalanced binary ECG classification task, and despite the common practice of reporting results based on accuracy in the papers related to foundation models, this metric is not suitable for my case. In fact, while I achieved an accuracy of 0.83 using SVM on MOMENT's embeddings, the F1-score was only 0.21. This is very low compared to a CNN-based approach that achieved an F1-score of 0.89. |
from peft import LoraConfig, get_peft_model
from momentfm import MOMENTPipeline
model = MOMENTPipeline.from_pretrained(
"/hy-tmp/better464/MOMENT-1-large",
model_kwargs={
'task_name': 'forecasting',
'forecast_horizon': 192,
'head_dropout': 0.1,
'weight_decay': 0,
'freeze_encoder': False, # Freeze the patch embedding layer
'freeze_embedder': False, # Freeze the transformer encoder
'freeze_head': False, # The linear forecasting head must be trained 否则输出会不一致
},
)
lora_config = LoraConfig(
r=64,
lora_alpha=32,
target_modules=["q", "v"],
lora_dropout=0.05,
)
model_new = get_peft_model(model, lora_config)
print('LoRA enabled')
model_new.print_trainable_parameters() File /usr/local/miniconda3/envs/moment/lib/python3.11/site-packages/accelerate/accelerator.py:34 ImportError: cannot import name 'split_torch_state_dict_into_shards' from 'huggingface_hub' (/usr/local/miniconda3/envs/moment/lib/python3.11/site-packages/huggingface_hub/init.py) QuestionsHow I can do fine-tuning the pre-trained model for forecasting using LoRA? I modified the code in the demo, but it doesn't seem to work well, thanks. (Python packages are all versions required in the requirements.) |
Hi all, Please let us know if you have any questions or concerns, and please don't hesitate to open another issue! Best, |
Hi, How I can do fine-tuning the pre-trained model with my dataset?
The text was updated successfully, but these errors were encountered: