About Saving Models

[misszhou966]

Hello, may I ask why I did not generate a file like denoisenet epoch {epoch: 02d} - val_loss {val_loss:. 6f}. ckpt in the logs folder after training。

[ddsediri]

Hi @misszhou966, the checkpoint should be generated within the log folder according to the filename below:

IterativePFN/train.py

Line 59 in 79efe40

filename='denoisenet-epoch{epoch:02d}-val_loss{val_loss:.6f}',

Also, checkpoints are saved every 5 epochs with the current settings, how many epochs have you trained for so far?

IterativePFN/train.py

Line 55 in 79efe40

every_n_epochs=args.save_interval,

[misszhou966]

Hello, I have trained 100 rounds now, but in the last round saved in the logs folder, there is only the version 0 folder and the run. log file in the DPUNet_XX folder. In the version 0 folder, there are only the hparams. yaml file and events. out. events file. I am not sure if this is related to my modification of the training set. Due to memory constraints, I modified the resolutions parameter to only train 10000 points of data.

[ddsediri]

Training only on the 10K data should be fine and I can run this code without any issues. Can you post a snippet of the train.py code to make sure the filename for the checkpoint is valid?

The DPUNet_XX folder should contain the checkpoints. What version of pytorch lightning are you using?

[misszhou966]

import os
import argparse
import torch
import torch.utils.tensorboard
from tqdm.auto import tqdm
from pytorch_lightning import Trainer, seed_everything
from pytorch_lightning.callbacks import ModelCheckpoint
from pytorch_lightning.loggers import TensorBoardLogger
import logging
from models.denoise import DenoiseNet
from utils.misc import get_log_dir_name_tblogger, seed_all, str_list

def main(args):

# Logging
local_rank = os.environ.get('LOCAL_RANK', 0)

if local_rank == 0:
    log_dir_name = get_log_dir_name_tblogger(name='D%s_' % (args.dataset))
    os.makedirs(os.path.join(args.log_root, log_dir_name))
    os.environ['LOG_DIR_NAME'] = log_dir_name
else:
    log_dir_name = os.environ['LOG_DIR_NAME']

log_dir = os.path.join(args.log_root, log_dir_name)

# configure logging on module level, redirect to file
logger = logging.getLogger('pytorch_lightning.core')
logger.addHandler(logging.FileHandler(os.path.join(log_dir, 'run.log')))

# Model
logger.info('INFO: Building model...')
model = DenoiseNet(args)

for k, v in vars(args).items():
    logger.info('[ARGS::%s] %s' % (k, repr(v)))

logger.info(repr(model))

# Main loop
try:
    logger.info('INFO: Start training...')
    seed_everything(args.seed, workers=True)
    trainer = Trainer(
        accelerator='gpu',
        devices=args.n_gpu,
        #accelerator='cpu',
        
        num_nodes=1,
        logger=TensorBoardLogger(args.log_root, name=log_dir_name),
        deterministic=True,
        max_epochs=100,
        check_val_every_n_epoch=args.save_interval,
        callbacks=[
                    ModelCheckpoint(
                         # monitor='val_loss',
                         every_n_epochs=args.save_interval, 
                         save_on_train_epoch_end=False,
                         save_top_k = -1,
                         dirpath=log_dir,
                         filename='denoisenet-epoch{epoch:02d}-val_loss{val_loss:.6f}',
                         auto_insert_metric_name=False
                    )
                    ],
        strategy="ddp"
    )

    trainer.fit(model)
except KeyboardInterrupt:
    logger.info('INFO: Terminating...')
    print('Terminating...')

if name == 'main':
# Arguments
parser = argparse.ArgumentParser()
## Dataset and loader
parser.add_argument('--dataset_root', type=str, default='./data')
parser.add_argument('--dataset', type=str, default='PUNet')
parser.add_argument('--changelog', type=str, default='')
parser.add_argument('--patches_per_shape_per_epoch', type=int, default=500)
parser.add_argument('--patch_ratio', type=float, default=1.2)
parser.add_argument('--resolutions', type=str_list, default=['10000_poisson'], choices=[['x0000_poisson'], ['x0000_poisson', 'y0000_poisson'], ['x0000_poisson', 'y0000_poisson', 'z0000_poisson']])
parser.add_argument('--noise_min', type=float, default=0.005)
parser.add_argument('--noise_max', type=float, default=0.02)
parser.add_argument('--train_batch_size', type=int, default=4)
parser.add_argument('--val_batch_size', type=int, default=4)
parser.add_argument('--n_gpu', type=int, default=8)
parser.add_argument('--num_workers', type=int, default=12)
parser.add_argument('--save_interval', type=int, default=5)
parser.add_argument('--debug', type=bool, default=False)
parser.add_argument('--aug_rotate', type=eval, default=True, choices=[True, False])
## Model architecture
parser.add_argument('--loss_type', type=str, default='NN', choices=['NN', 'NN_no_stitching'])
## Optimizer and scheduler
parser.add_argument('--sched_patience', default=10, type=int)
parser.add_argument('--sched_factor', default=0.5, type=float)
parser.add_argument('--min_lr', default=1e-7, type=float)
parser.add_argument('--lr', type=float, default=1e-4)
parser.add_argument('--weight_decay', type=float, default=0)
## Training
parser.add_argument('--seed', type=int, default=2020)
parser.add_argument('--log_root', type=str, default='./logs')
parser.add_argument('--device', type=str, default='cuda')
parser.add_argument('--val_noise', type=float, default=0.015)

# Ablation parameters
parser.add_argument('--patch_size', type=int, default=500)
parser.add_argument('--frame_knn', type=int, default=32) # Neighbourhood side for graph convolution
parser.add_argument('--num_modules', type=int, default=4)
parser.add_argument('--noise_decay', type=int, default=4) # Noise decay is set to 16/T where T=num_modules or set to 1 for no decay

args = parser.parse_args()

main(args)

[misszhou966]

My pytorch lightning is 1.7.6

[ddsediri]

You are using a Linux setup, right? I've not been able to reproduce this issue and checkpointing always occurs. Can you share the run.log file? Do you get validation output?

Also,, could you try rerunning the train script with a simple checkpoint name such as filename='denoisenet'? After 5 epochs, you should see a checkpoint which will be overwritten every 5 epochs. You can reduce the patches_per_shape_per_epoch to a small value so that the training will run faster and you should see if checkpoints are saved or not.

[misszhou966]

Yes,This is my run.log file

INFO: Building model...
[ARGS::dataset_root] './data'
[ARGS::dataset] 'PUNet'
[ARGS::changelog] ''
[ARGS::patches_per_shape_per_epoch] 1000
[ARGS::patch_ratio] 1.2
[ARGS::resolutions] ['10000_poisson']
[ARGS::noise_min] 0.005
[ARGS::noise_max] 0.02
[ARGS::train_batch_size] 4
[ARGS::val_batch_size] 4
[ARGS::n_gpu] 1
[ARGS::num_workers] 12
[ARGS::save_interval] 5
[ARGS::debug] False
[ARGS::aug_rotate] True
[ARGS::loss_type] 'NN'
[ARGS::sched_patience] 10
[ARGS::sched_factor] 0.5
[ARGS::min_lr] 1e-07
[ARGS::lr] 0.0001
[ARGS::weight_decay] 0
[ARGS::seed] 2020
[ARGS::log_root] './logs'
[ARGS::device] 'cuda'
[ARGS::val_noise] 0.015
[ARGS::patch_size] 500
[ARGS::frame_knn] 32
[ARGS::num_modules] 4
[ARGS::noise_decay] 4
DenoiseNet2(
(feature_nets): ModuleList(
(0): FeatureExtraction(
(conv1): DynamicEdgeConv()
(conv2): DynamicEdgeConv()
(conv3): DynamicEdgeConv()
(conv4): DynamicEdgeConv()
(linear1): Linear(in_features=512, out_features=256, bias=False)
(linear2): Linear(in_features=256, out_features=128, bias=True)
(linear3): Linear(in_features=128, out_features=3, bias=True)
)
(1): FeatureExtraction(
(conv1): DynamicEdgeConv()
(conv2): DynamicEdgeConv()
(conv3): DynamicEdgeConv()
(conv4): DynamicEdgeConv()
(linear1): Linear(in_features=512, out_features=256, bias=False)
(linear2): Linear(in_features=256, out_features=128, bias=True)
(linear3): Linear(in_features=128, out_features=3, bias=True)
)
(2): FeatureExtraction(
(conv1): DynamicEdgeConv()
(conv2): DynamicEdgeConv()
(conv3): DynamicEdgeConv()
(conv4): DynamicEdgeConv()
(linear1): Linear(in_features=512, out_features=256, bias=False)
(linear2): Linear(in_features=256, out_features=128, bias=True)
(linear3): Linear(in_features=128, out_features=3, bias=True)
)
(3): FeatureExtraction(
(conv1): DynamicEdgeConv()
(conv2): DynamicEdgeConv()
(conv3): DynamicEdgeConv()
(conv4): DynamicEdgeConv()
(linear1): Linear(in_features=512, out_features=256, bias=False)
(linear2): Linear(in_features=256, out_features=128, bias=True)
(linear3): Linear(in_features=128, out_features=3, bias=True)
)
)
)
INFO: Start training...
INFO: Current epoch validation loss: 0.007462

[misszhou966]

And I'm sorry that I don't quite understand your statement: Do you get validation output

[misszhou966]

Yes, I am using Linux setup and I would like to follow your suggestion: try rerunning the train script with a simple checkpoint name such as file name='denoisenet '. However, the GPU of the server I am using is currently being occupied by someone else, so I cannot try immediately.

[ddsediri]

That's OK, what I meant was, in the run.log file, are you getting any validation losses recorded. In the run.log output you shared, the epoch training loss and validation loss are missing, can you please add that?

After the "INFO: Start training..." line, the output should be something like (please note for this output I am using n_gpu=3 so the training and validation losses are calculated for each gpu):

INFO: Start training...
INFO: Current epoch validation loss: 0.009210
INFO: Current epoch validation loss: 0.009114
INFO: Current epoch validation loss: 0.009084
INFO: Current epoch training loss: 0.094350
INFO: Current epoch training loss: 0.094320
INFO: Current epoch training loss: 0.094261
INFO: Current epoch training loss: 0.038857
INFO: Current epoch training loss: 0.038838
INFO: Current epoch training loss: 0.038840
INFO: Current epoch training loss: 0.011389
INFO: Current epoch training loss: 0.011371
INFO: Current epoch training loss: 0.011371
INFO: Current epoch training loss: 0.007205
INFO: Current epoch training loss: 0.007202
INFO: Current epoch training loss: 0.007211
INFO: Current epoch validation loss: 0.000672
INFO: Current epoch validation loss: 0.000712
INFO: Current epoch validation loss: 0.000790
INFO: Current epoch training loss: 0.004360
INFO: Current epoch training loss: 0.004367
INFO: Current epoch training loss: 0.004368
INFO: Current epoch training loss: 0.002468
INFO: Current epoch training loss: 0.002467
INFO: Current epoch training loss: 0.002467
INFO: Current epoch training loss: 0.001674
INFO: Current epoch training loss: 0.001677
INFO: Current epoch training loss: 0.001673
INFO: Current epoch training loss: 0.001200
INFO: Current epoch training loss: 0.001202
INFO: Current epoch training loss: 0.001203
INFO: Current epoch training loss: 0.000902
INFO: Current epoch training loss: 0.000903
INFO: Current epoch training loss: 0.000907
INFO: Current epoch validation loss: 0.000326
INFO: Current epoch validation loss: 0.000385
INFO: Current epoch validation loss: 0.000376
INFO: Current epoch training loss: 0.000730
INFO: Current epoch training loss: 0.000733
INFO: Current epoch training loss: 0.000735
INFO: Current epoch training loss: 0.000654
INFO: Current epoch training loss: 0.000655
INFO: Current epoch training loss: 0.000653
INFO: Current epoch training loss: 0.000621
INFO: Current epoch training loss: 0.000619
INFO: Current epoch training loss: 0.000625
INFO: Current epoch training loss: 0.000594
INFO: Current epoch training loss: 0.000600
INFO: Current epoch training loss: 0.000600
INFO: Current epoch training loss: 0.000581
INFO: Current epoch training loss: 0.000580
INFO: Current epoch training loss: 0.000579
INFO: Current epoch validation loss: 0.000292
INFO: Current epoch validation loss: 0.000336
INFO: Current epoch validation loss: 0.000345
INFO: Current epoch training loss: 0.000561
INFO: Current epoch training loss: 0.000559
INFO: Current epoch training loss: 0.000564
INFO: Current epoch training loss: 0.000550
INFO: Current epoch training loss: 0.000549

And so on...

[misszhou966]

This is the partial display of the terminal in my last training, I'm not sure if you're referring to this
Epoch 95: 100%|███████████████████████████████████████████| 5000/5000 [36:16<00:00, 2.30it/s, loss=0.000443, v_num=0]INFO: Current epoch training loss: 0.000482
Epoch 96: 100%|███████████████████████████████████████████| 5000/5000 [36:32<00:00, 2.28it/s, loss=0.000471, v_num=0]INFO: Current epoch training loss: 0.000480
Epoch 97: 100%|███████████████████████████████████████████| 5000/5000 [36:18<00:00, 2.30it/s, loss=0.000518, v_num=0]INFO: Current epoch training loss: 0.000481
Epoch 98: 100%|███████████████████████████████████████████| 5000/5000 [36:36<00:00, 2.28it/s, loss=0.000452, v_num=0]INFO: Current epoch training loss: 0.000481
Epoch 99: 100%|████████████████████████████████████████████| 5005/5005 [42:56<00:00, 1.94it/s, loss=0.00051, v_num=0]INFO: Current epoch validation loss: 0.000271███████████████████████████████████████████| 5/5 [06:29<00:00, 77.87s/it]
Epoch 99: 100%|████████████████████████████████████████████| 5005/5005 [42:56<00:00, 1.94it/s, loss=0.00051, v_num=0INFO: Current epoch training loss: 0.000480
Epoch 99: 100%|████████████████████████████████████████████| 5005/5005 [42:57<00:00, 1.94it/s, loss=0.00051, v_num=0]Trainer.fit stopped: max_epochs=100 reached.
Epoch 99: 100%|████████████████████████████████████████████| 5005/5005 [42:57<00:00, 1.94it/s, loss=0.00051,

[ddsediri]

So the terminal output seems to show the script working as intended. The training epochs are completed and the validation occurs every 5th epoch. However, my last comment was about the run.log which is in the ./logs/DPUNet_xxx folder.

The run.log file you previously attached was incomplete (it just showed the sanity validation step after training had started) and I am just wondering if there is an issue with writing to that folder... The checkpoints should also be saved in ./logs/DPUNet_xxx.

[misszhou966]

Based on what you mean, there may have been an issue during the writing process. You mentioned that the run.log file I provided was incomplete, but I have checked all the run.log files I generated and found that their content is the same as what I provided you with, without any further information.

[ddsediri]

Ah ok, so it might be an issue there. When you have access to the server again, try retraining and see if the run.log file is updated with the training and validation losses (similar to my above comment). Also, you can simplify the name of the checkpoint file.

Without knowing the details of the server you are running the script on, it would be difficult to troubleshoot. If you had access to a workstation, you could adjust the network size (by making the network layers smaller) and try to train on the workstation which might make debugging more efficient. Also, you might try running some simpler examples of pytorch lightning scripts on the server to see if writing to the log directory works as intended.

I also did a bit more digging into the TensorboardLogger. Can you check if you have the following installed to make sure any lightning dependencies are not missing:
tensorboardX==2.5.1
fsspec==2022.8.2

[misszhou966]

Okay, thank you very much for your patient answer. I will try again and check the dependencies you mentioned.