Question about Yarn environment configuration (v2)

[Yiyi-philosophy]

Hi Yarn team,

I hope this issue finds you well. I clone your git code (v2, 2weeks ago) in our machine and found mistakes:

Traceback (most recent call last):
  File "/app/yarn_4/finetune.py", line 293, in <module>
    main(args.parse_args())
  File "/app/yarn_4/finetune.py", line 52, in main
    from scaled_rope.modeling_llama_yarn import LlamaForCausalLM
  File "/app/yarn_4/scaled_rope/modeling_llama_yarn.py", line 34, in <module>
    from transformers.utils import (
ImportError: cannot import name 'is_flash_attn_2_available' from 'transformers.utils' (/opt/conda/lib/python3.10/site-pack
ages/transformers/utils/__init__.py)

In our current environment, we are using the following versions:

• Python: 3.10
• PyTorch: 2.1.0
• CUDA: 11.8
• Transformers: 4.34.0
• PyTorch-CUDA: 11.8
• Torchtriton: 2.1.0
• Torchvision: 0.16.0
• Accelerate: 0.24.1
• Deepspeed: 0.12.3
• flash-attn: 2.3.3

We are interested in fine-tuning the Yarn environment for our specific setup. Specifically, we would like to inquire about the versions of transformers, accelerate and deepspeed used in the Yarn environment. Could you please provide details on how these tools are configured in your environment?

Any guidance or information you can offer regarding this matter would be greatly appreciated.

Thank you for your time and assistance!

[cebtenzzre]

Based on that import, it appears that the minimum supported version of transformers is actually v4.35.0.

[Yiyi-philosophy]

Yes, I appreciate your input. I did attempt to use Transformers v4.35.0, but encountered the following error:

/opt/conda/lib/python3.10/site-packages/datasets/table.py:1421: FutureWarning: promote has been superseded by mode='
default'.                                                                                                           
  table = cls._concat_blocks(blocks, axis=0)                                                                        
Traceback (most recent call last):                                                                                  
  File "/app/yarn_4/finetune.py", line 295, in <module>                                                             
    main(args.parse_args())                                                                                         
  File "/app/yarn_4/finetune.py", line 158, in main                                                                 
    #     model.gradient_checkpointing_enable()                                                                     
  File "/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1872, in gradient_checkpointin
g_enable                                                                                                            
    self._set_gradient_checkpointing(enable=True, gradient_checkpointing_func=gradient_checkpointing_func)          
TypeError: LlamaPreTrainedModel._set_gradient_checkpointing() got an unexpected keyword argument 'enable'

Further investigation led me to related GitHub issues:

• huggingface/transformers#27596
• hiyouga/LLaMA-Factory#1391

According to these discussions, it's suggested that using Transformers v4.34.0 resolves the issue. I wanted to share this additional information for context.

Besides, I can change is_flash_attn_2_available to another implemtation to fix problem:
at here
change to

from transformers.utils import (
    add_start_docstrings,
    add_start_docstrings_to_model_forward,
    # is_flash_attn_2_available,
    logging,
    replace_return_docstrings,
)
from .configuration_llama import LlamaConfig

# if is_flash_attn_2_available():
try: 
    from flash_attn import flash_attn_func, flash_attn_varlen_func
    from flash_attn.bert_padding import index_first_axis, pad_input, unpad_input  # noqa
except:
    print("Error is_flash_attn_2_available")

[Yiyi-philosophy]

But at the same time, if I want to use --deepspeed, something went wrong:

Traceback (most recent call last):
  File "/app/yarn_4/finetune.py", line 295, in <module>                                                             
    main(args.parse_args())
  File "/app/yarn_4/finetune.py", line 160, in main                                                                 
    accelerator.register_for_checkpointing(scheduler)
  File "/opt/conda/lib/python3.10/site-packages/accelerate/accelerator.py", line 3139, in register_for_checkpointing
    raise ValueError(err)                                                                                           
ValueError: All `objects` must include a `state_dict` and `load_state_dict` function to be stored. The following inputs are invalid:                                                                                                    
        - Item at index 0, `DummyScheduler`  
        -

Further investigation led me to related GitHub issues:

nomic-ai/gpt4all#233

How can I fix this problem?
(this time Transformers is still 4.34.0)

[jquesnelle]

Yes, I appreciate your input. I did attempt to use Transformers v4.35.0, but encountered the following error:

/opt/conda/lib/python3.10/site-packages/datasets/table.py:1421: FutureWarning: promote has been superseded by mode='
default'.                                                                                                           
  table = cls._concat_blocks(blocks, axis=0)                                                                        
Traceback (most recent call last):                                                                                  
  File "/app/yarn_4/finetune.py", line 295, in <module>                                                             
    main(args.parse_args())                                                                                         
  File "/app/yarn_4/finetune.py", line 158, in main                                                                 
    #     model.gradient_checkpointing_enable()                                                                     
  File "/opt/conda/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1872, in gradient_checkpointin
g_enable                                                                                                            
    self._set_gradient_checkpointing(enable=True, gradient_checkpointing_func=gradient_checkpointing_func)          
TypeError: LlamaPreTrainedModel._set_gradient_checkpointing() got an unexpected keyword argument 'enable'

This was caused by a recent transformers change -- I've pushed cdb6459 to fix this!