trainer doesn't save any checkpoint when using checkpoint_callback

[allenhung1025]

Environment

• OS: Ubuntu 18.04.5 LTS
• python3.7.11
• pytorch 1.11.0+cu102
• pytorch lightning 1.6.1

Code

• validation_step

def validation_step(self, batch, batch_idx):
        data, aug_data, label = batch
        input_ids, attention_mask, labels = data['input_ids'], data['attention_mask'], label
        aug_input_ids, aug_attention_mask = aug_data['input_ids'], aug_data['attention_mask']
        input_ids, attention_mask, labels = input_ids.cuda(), attention_mask.cuda(), labels.cuda()
        aug_input_ids, aug_attention_mask = aug_input_ids.cuda(), aug_attention_mask.cuda()

        output = self.compute_loss_prediction_and_metrics(input_ids, aug_input_ids, attention_mask, aug_attention_mask, labels)
        loss, contrastive_loss, poincare_loss, valid_pred, valid_label, valid_acc, valid_micro_f1, valid_macro_f1, valid_weighted_f1 = output

        self.valid_pred.extend(valid_pred)
        self.valid_label.extend(valid_label)

        self.log('valid_acc', valid_acc, on_step = False, on_epoch = True)
        self.log('valid_micro_f1', valid_micro_f1)
        self.log('valid_macro_f1', valid_macro_f1)
        self.log('valid_weighted_f1', valid_weighted_f1)
        
        return {'valid_loss': loss, 'valid_pred': valid_pred, 'valid_acc': valid_acc,
                'valid_micro_f1': valid_micro_f1, 'valid_macro_f1': valid_macro_f1, 'valid_weighted_f1': valid_weighted_f1}

• validation_epoch_end

def validation_epoch_end(self , outs):
        valid_acc, valid_micro_f1, valid_macro_f1, valid_weighted_f1 = self.acc_and_f1_score(self.valid_pred, self.valid_label)
        self.valid_pred = []
        self.valid_label = []
        self.log('valid_acc_epoch_end', valid_acc)
        self.log('valid_micro_f1_epoch_end', valid_micro_f1)
        self.log('valid_macro_f1_epoch_end', valid_macro_f1)
        self.log('valid_weighted_f1_epoch_end', valid_weighted_f1)

        return {'val_loss': outs[0]['valid_loss'], 'valid_acc_epoch_end': valid_acc, 'valid_micro_f1_epoch_end':valid_micro_f1,
                'valid_macro_f1_epoch_end': valid_macro_f1, 'valid_weighted_f1_epoch_end':valid_weighted_f1}

• trainer

checkpoint_callback = ModelCheckpoint(dirpath = args.save_path, 
                                      verbose=True,
                                      save_top_k = -1,
                                      save_last=True)
 trainer = pl.Trainer(gpus = 1,
                     callbacks = [checkpoint_callback],
                     logger=logger,
                     default_root_dir=args.save_path,
                     max_epochs = args.max_epochs)
 trainer.fit(gm)

Expected behavior

• save a model for each epoch under args.save_path

Real behavior

• No model is saved under args.save_path
• only log the loss, accuracy, ..., etc.
  

Request for help

• Does anyone can point me out where I got wrong? I have searched on Google, looked the document thoroughly, and still couldn't find a solution. Thanks in advance.

[rohitgr7]

can you check the actual dirpath of the checkpoint callback using trainer.checkpoint_callback.dirpath and see if there are any checkpoints there?

if they are still now showing up, can you share a reproducible minimal example using BoringModel?

[allenhung1025]

Hi @rohitgr7,

When I use BoringMode with model_checkpoint and trainer that I defined, the tariner can save the checkpoint successfully.

However, when I switched back to the model I defined, the trainer wouldn't save any checkpoint under trainer.checkpoint_callback.dirpath. The logging worked fine, though.

Thanks for your kind reply.

[rohitgr7]

checkpoint_callback = ModelCheckpoint(dirpath = args.save_path, 
                                      verbose=True,
                                      save_top_k = -1,
                                      save_last=True)

since you are not using any monitor, it should just checkpoint after every epoch.

do you see any warnings being emitted from checkpoint callback during the run?

[allenhung1025]

The cloest warning related to callback is stating that the directory exists.

/home/allenhung/miniconda3/envs/finegrained/lib/python3.7/site-packages/pytorch_lightning/callbacks/model_checkpoint.py:608: UserWarning: Checkpoint directory /home/allenhung/fine-grained-classification-$
epro/checkpoint_GE_4 exists and is not empty.
  rank_zero_warn(f"Checkpoint directory {dirpath} exists and is not empty.")

[rohitgr7]

so it's showing the directory is not empty and not even storing anything in that. this is weird.
not sure if there is some issue in your file-system, but I'd need a reproducible script to check the issue.