馃殌 Feature request
When training or fine-tuning a transformer model, people usually warmup for 10% training steps. For now, transformers only provide the parameter of warmup_steps. A warmup_ratio parameter can be helpful, it means warmup for some percentage of total training steps.
Motivation
To use the parameter of warmup_steps, people need to know the total training steps. When people use the training epoch parameter instead of max_steps, it is hard to know the total training steps. A warmup_ratio parameter get rid of people knowing total training steps. Another reason for using warmup_ratio parameter is it can help people write less hard code. People have different total training steps for different dataset, but people usually set warmup_ratio as 10% as default.
Original Usage may like this:
python run_ner.py --data_dir some_data_dir \ --model_name_or_path some_model \ --output_dir some_output_dir \ --max_seq_length 512 \ --num_train_epochs 10 \ --warmup_steps 35 \ --per_device_train_batch_size 8 \ --do_train \
New usage may like this:
python run_ner.py --data_dir some_data_dir \ --model_name_or_path some_model \ --output_dir some_output_dir \ --max_seq_length 512 \ --num_train_epochs 10 \ --warmup_ratio 0.1 \ --per_device_train_batch_size 8 \ --do_train \
Also, we can merge warmup_step and warmup_ratio into one parameter. If user input a number 0 <= x < 1, it will be considered as warmup_ratio. If user input an interger, it will be considered as warmup_step.
Your contribution
I can submit a PR to complete this feature. If similar feature is alreadly in this repo, please just close this issue.
馃殌 Feature request
When training or fine-tuning a transformer model, people usually warmup for 10% training steps. For now, transformers only provide the parameter of warmup_steps. A warmup_ratio parameter can be helpful, it means warmup for some percentage of total training steps.
Motivation
To use the parameter of warmup_steps, people need to know the total training steps. When people use the training epoch parameter instead of max_steps, it is hard to know the total training steps. A warmup_ratio parameter get rid of people knowing total training steps. Another reason for using warmup_ratio parameter is it can help people write less hard code. People have different total training steps for different dataset, but people usually set warmup_ratio as 10% as default.
Original Usage may like this:
python run_ner.py --data_dir some_data_dir \ --model_name_or_path some_model \ --output_dir some_output_dir \ --max_seq_length 512 \ --num_train_epochs 10 \ --warmup_steps 35 \ --per_device_train_batch_size 8 \ --do_train \New usage may like this:
python run_ner.py --data_dir some_data_dir \ --model_name_or_path some_model \ --output_dir some_output_dir \ --max_seq_length 512 \ --num_train_epochs 10 \ --warmup_ratio 0.1 \ --per_device_train_batch_size 8 \ --do_train \Also, we can merge warmup_step and warmup_ratio into one parameter. If user input a number 0 <= x < 1, it will be considered as warmup_ratio. If user input an interger, it will be considered as warmup_step.
Your contribution
I can submit a PR to complete this feature. If similar feature is alreadly in this repo, please just close this issue.