i use transformer's bitnet to train, and find: after reaching a certain number of steps, acc will drop sharply to low value

[Airthrix]

i use transformer's bitnet to train, and find: after reaching a certain number of steps, acc will drop sharply to low value

Initially, the loss will decrease normally, and the acc will also rise normally to around 0.18，but
Once the loss drops to around 0.6, the acc will suddenly start to drop sharply to a very low level.

I used the bitnet framework for training and attempted to train on both cpu and gpu. However, I found that the acc would must drop sharply after a certain number of steps.
What could be the reason for this？ haaa

[buaahsh]

Could you provide more details, such as the task type, training sequence length, and learning rate? Additionally, have you considered reducing the learning rate?

[Airthrix]

My learning rate: 2.6e-5 and 1.0e-5
task type: are simple dialogue
sequence length: 10~20 token, very short
(i deliberately used this short dataset for rapid training to see the effect of the Bitnet framework)

I have tried to reduce the learning rate, but the nacc still dropped sharply to 0 with more steps, and the result remained the same

[buaahsh]

I do not observe any obvious issues. Which fine-tuning framework are you using? I can provide a minimal implementation using TRL.

install the necessary packages:

pip install trl
pip install git+https://github.com/shumingma/transformers.git

a sample code snippet (Please note that it is only a minimal example—hyperparameters such as batch size and learning rate should be tuned for optimal performance):

from trl import SFTConfig, SFTTrainer
from datasets import load_dataset

dataset = load_dataset("trl-lib/Capybara", split="train")

training_args = SFTConfig(
    max_length=2048,
    output_dir="/tmp",
    per_device_train_batch_size=4,
)

trainer = SFTTrainer(
    model="microsoft/bitnet-b1.58-2B-4T-bf16",
    train_dataset=dataset,
    args=training_args,
)

trainer.train()

Below is our loss curve, the loss remains stable and does not exhibit signs of divergence:

[Airthrix]

I do not observe any obvious issues. Which fine-tuning framework are you using? I can provide a minimal implementation using TRL.

install the necessary packages:

pip install trl
pip install git+https://github.com/shumingma/transformers.git

a sample code snippet (Please note that it is only a minimal example—hyperparameters such as batch size and learning rate should be tuned for optimal performance):

from trl import SFTConfig, SFTTrainer
from datasets import load_dataset

dataset = load_dataset("trl-lib/Capybara", split="train")

training_args = SFTConfig(
max_length=2048,
output_dir="/tmp",
per_device_train_batch_size=4,
)

trainer = SFTTrainer(
model="microsoft/bitnet-b1.58-2B-4T-bf16",
train_dataset=dataset,
args=training_args,
)

trainer.train()
Below is our loss curve, the loss remains stable and does not exhibit signs of divergence:

I started training from scratch. I didn't use fine-tuning because I knew it would work 100% with fine-tuning.

I started the training from scratch. I did it to verify the performance of this framework. My bitnet framework is directly called from the transformer. I carefully checked that all the Tokenizers were correct, and I found that the code structure of BitNet (such as the BitNetRMSNorm, BitNetRotaryEmbedding and other modules) is highly similar to the LLaMA series of models. LLaMA uses the SentencePiece tokenizer (based on the sentencepiece library), so BitNet also follows this design.

No matter what, the training code, dataset and tokenizer on my side are all normal i sure.
The current problem remains: when training from scratch, I find that once the loss of bitnet approaches 0.6, the acc will drop sharply to 0, which is very strange. I also modified the bitnet framework, but it was still the same situation

[Airthrix]

I have found the reason. It's my own problem. It has been solved ^^

I verified the performance of the Bitnet framework. It is relatively stable and when used for embedded robot interaction, its power consumption is one-tenth of that of the normal model.

In the future, if hardware chips support ternary, it will further reduce power consumption, training and inference time