The speed issue of validation stage when run two or more same model in different datasets

[CKK-coder]

when I use different GPU servers to train adaface in different daatsets, the speed of train stage is normal. But when this two task run validation stage in the same time, the cpu utilization is very low and the "validation dataloader" will take so long time. Specifically, only one task running, it will take about 10 min or less to "validation dataloader" after one train epoch. When two task running, it will task more than several hours to "validation dataloader" after one train epoch. What is the reason of this issue, how can i sovle it? Look forward to your reply !