how to checkpoint all the validation scores in huggingface trainer

Hi
I want to find the best model per evaluation score. Could you please give me more info, how I can checkpoint all evaluation scores in each step of training to find the best model? thanks

[yezhengli-Mr9]

Interested in this as well. Do not find a solution from blog "How to monitor both train and validation metrics at the same step?".

[yezhengli-Mr9]

Hi
I want to find the best model per evaluation score. Could you please give me more info, how I can checkpoint all evaluation scores in each step of training to find the best model? thanks

I think I figure it out:

training_args = TrainingArguments(
    output_dir='./results',          # output directory
    num_train_epochs=3,              # total number of training epochs
    per_device_train_batch_size=16,  # batch size per device during training
    per_device_eval_batch_size=64,   # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs',            # directory for storing logs
    logging_steps=100,
++    evaluation_strategy='steps',
)

[yezhengli-Mr9]

Interested in this as well. Do not find a solution from blog "How to monitor both train and validation metrics at the same step?".

Hi
I want to find the best model per evaluation score. Could you please give me more info, how I can checkpoint all evaluation scores in each step of training to find the best model? thanks

Correspondingly, I put

# https://huggingface.co/transformers/training.html
#metric = load_metric('glue', 'mrpc')
def compute_metrics(p):#: EvalPrediction
# def compute_metrics(p):
    preds = p.predictions[0] if isinstance(p.predictions, tuple) else p.predictions
#     preds = np.squeeze(preds) if is_regression else np.argmax(preds, axis=1)
    preds = np.argmax(preds, axis=1)
#     if data_args.task_name is not None:
#         result = metric.compute(predictions=preds, references=p.label_ids)
#         if len(result) > 1:
#             result["combined_score"] = np.mean(list(result.values())).item()
#         return result
#     elif is_regression:
#         return {"mse": ((preds - p.label_ids) ** 2).mean().item()}
#     else:
    return {"accuracy": (preds == p.label_ids).astype(np.float32).mean().item()}

trainer = Trainer(
    model=model,                         # the instantiated 🤗 Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_dataset,         # training dataset
    eval_dataset=val_dataset,             # evaluation dataset
++    compute_metrics = compute_metrics
)

trainer.train()

and result in output:

[yezhengli-Mr9]

• However, in terms of Accuracy, Not for sure it is on training dataset or validation dataset.

[patil-suraj]

Accuracy here is for validation dataset.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.