dataloader is so slow?

[ghoshaw]

Hi, I write a dataloader class as customdata, and I tried when the train data is small the training seems ok. But when the dataset of train goes to 600K, then the log not moving after "mmseg - INFO - Use load_from_openmmlab loader".
after 12h, log is still the same.

According to my understanding, the dataloader only load the filenames of train dataset to the memory and then read the image when train is started. It should not need so much time. Is that right? and how to solve this problem?
num worker per gpu=4 or 0 seems to be useless about this question.

[Junjun2016]

Hi @ghoshaw
Have you solved the problem?
600K means the number of training samples is 600K?
And could you please provide the whole config and log files.

[ghoshaw]

Hi, here is my log and config, and my dataloader upload either. my data format is a txt, include "XXX.jpg,XXX.png",so I split with the sign ",".
log:
20210709_143000.log
config:
ocrnet_hr48_512x512_80k_coce.txt
dataloader code:
mydata.txt
cocedata.txt
and base, dataconfig:
coce.txt

[Junjun2016]

Hi @ghoshaw
You have passed the wrong args.
You may refer to our customize_datasets and other tutorials.

[ghoshaw]

@Junjun2016 ,Hi, I did not understand. I change the data reading code, and delete the data_root and ann_dir in customdata, I directly read the image name from a list txt which passed by img_dir. By the way, my list txt include the full path of image and mask.

[ghoshaw]

Hi, again. I tried to use one gpu instead of two gpus, then it can run success which is wired. Do you have any idea why multi-gpu can not run?
By the way, when I control+c to stop the run of mulit-gpu, it says waitting pid somthing.

[Junjun2016]

Can you run our latest version with multi-gpu?

[ghoshaw]

I tried, and same result. Only support one gpu.

[Junjun2016]

How about our official supported datasets?

[ghoshaw]

you mean run with your official data load code with your supported datafile format ? I will give it a try.

[Junjun2016]

Yes, have a try.

[ghoshaw]

unfortunately, it still can not run with the source dataloader with muligpu. so I guess the question is with my machine?
By the way, when I control+c to stop the run, the log is :

[Junjun2016]

Follow the tutorials and run with the official supported datasets.

[clownrat6]

Hi @ghoshaw,
Can you provide the repo branch?
We will try to reproduce your error and try to fix it.

[ghoshaw]

git log info:
commit f5be6e2 (HEAD -> master, origin/master, origin/HEAD)
Author: BigDong yudongwang1226@gmail.com
Date: Thu Jul 8 19:14:56 2021 +0800

[Fix] fix url error in config docs (#680)

* [Fix] fix url error in config docs

* Fix URL

* Fix URL