Lora implementation in finetuning and evaluation #638
Conversation
There was a problem hiding this comment.
@anwai98 This is the script I used for finetuning on covid_if data. I refer to this one in the data I sent as 'my impl'.
| joint_model_params.append(params) | ||
|
|
||
| optimizer = torch.optim.Adam(joint_model_params, lr=1e-5) | ||
| scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="min", factor=0.9, patience=10) |
There was a problem hiding this comment.
@anwai98 The patience for the learning rate scheduler was set to 10 here but to 3 in the 'Resource Efficient Impl' - Could that be the reason for the performance difference ?
There was a problem hiding this comment.
There are a few more changes. We had a lower learning rate and a different optimizer as well (https://github.com/caroteu/micro-sam/blob/dcd6ecdc5ef600e07670db27ccfba54e81f156f7/finetuning/specialists/resource-efficient/covid_if_finetuning.py#L135-L136)
And probably 100 epochs in the other experiments might translate to a bit more than 10k iterations here.
There was a problem hiding this comment.
Potentially that could lead to a bit of a performance difference (unless it's a very severe drop in performance, that would be a different discussion)
There was a problem hiding this comment.
The optimizer is actually the same - I ran this here with Adam and 1e-5 too to make it consistent with the other workflow. The results I send you both have adam and lr=1e-5
No description provided.