AttributeError: 'LlamaRotaryEmbedding' object has no attribute 'cos_cached'

[Ronnie-Leon76]

I'm trying to fine-tune unsloth/yi-6b-bnb-4bit on a custom dataset but as soon I initiate the training using trainer.train wrapped in an learning_rate scheduler optimizer using CosineAnnealing learning rate scheduler I get the error shown in the image. The error "AttributeError: 'LlamaRotaryEmbedding' object has no attribute 'cos_cached'" seems to be related to the LLaMA model of the Transformers library. It appears that the issue is with the implementation of the rotary embedding, specifically in the rotate_half function, where the slicing should have been interleaved. I'll appreciate help solving the issue above.

[danielhanchen]

@Ronnie-Leon76 I'll check this out today!! Sorry on the issue!

[Ronnie-Leon76]

I'll really appreciate it

[hbernie]

Having the same issue with unsloth/llama-2-7b-chat-bnb-4bit

[danielhanchen]

@Ronnie-Leon76 @hbernie Apologies could not look into it yesterday - was a bit inundated with stuff :(( Will 100% do it today! :) Sorry again!

[Ronnie-Leon76]

It's okay. Is it something I can help with? Could you give me a high level breakdown of what needs to be done to fix the issue? Should we refine the rotate_half() method?

[danielhanchen]

@Ronnie-Leon76 @hbernie
I found the issue! A new push a few days ago was made to the latest HuggingFace transformers branch huggingface/transformers#27931 which broke it!!

I'm assuming you used the HF notebooks which we shared - for now I would comment out !pip install "git+https://github.com/huggingface/transformers.git" - we will install the correct version anyways.

Also if you're on a local PC, I would downgrade transformers via !pip install transformers==4.37.2

I will for now edit my notebooks to remove the lines - hope the temporary fix solves it!

[danielhanchen]

@Ronnie-Leon76 @hbernie I think I fixed it!! Hope you all can try it out :) I also updated all the notebooks on our HuggingFace branch https://huggingface.co/datasets/unsloth/notebooks/tree/main and on our blog posts.

No need to change your notebooks!
If you're using a local machine (Colab no need to), upgrade Unsloth via pip install --upgrade --force-reinstall --no-cache-dir git+https://github.com/unslothai/unsloth.git

Hope it works!

[Ronnie-Leon76]

@danielhanchen It works fine. Thanks a lot.