Validation too slow

[SuTanTank]

https://github.com/open-mmlab/mmediting/blob/1735f5c97172a2f33af76e8c3e65e891bda593f0/mmedit/models/base_models/base_edit_model.py#L158-L164

[LeoXing1996]

Hey @SuTanTank, thanks for your interest. MMEditing calculates low-level-related metrics (e.g., SSIM, PSNR, NIQE) with numpy implementation. Therefore, we move outputs to cpu in forward_inferencer.

[SuTanTank]

Thanks for your reply. @LeoXing1996
I just found the validation very slow and checked the code and thought it might be due to the 'cpu' thing. But after I removed the to('cpu'), the validation is still very slow. Any suggestions?

[LeoXing1996]

@SuTanTank Can you provide the config, the command you run, and the output log?

[SuTanTank]

Sorry it not the model and dataset inside mmedit. Here are the dataloader config, which is almost the same as training.

val_dataloader = dict(
    num_workers=8, # 1
    batch_size=24, # 1
    persistent_workers=False,
    drop_last=True,
    sampler=dict(type='DefaultSampler', shuffle=False),
    dataset=dict(
        type='ISOVideoDataset',
        metainfo=dict(dataset_type='XXX', task_name='XXX'),
        data_root='./data/XXX',
        data_prefix=dict(
            img='lq',
            gt='gt',
        ),
        pipeline=train_pipeline,
        num_input_frames=4,
        fixed_seq_len=100,
    ),
)

[LeoXing1996]

data_time denotes the time used to load data from dataloader. Maybe you should check your dataset in the validation phase. 🤔

[SuTanTank]

🤔 But data_time of training is quite normal. Maybe its an mmengine issue?

[LeoXing1996]

Can you try to set persistent_workers =True in your config?

[SuTanTank]

Tried, but no improvement. I will keep looking for the reason and update in this thread later.

[SuTanTank]

update: The major cause of slow validation is the visualization defined in the default_runtime.py. Disable it, and validation does run a little bit slower than training but I believe it's because of the metric calculation, which is acceptable.

before

after