Training problem

[rmmal]

I have a dataset that has 12982 images
when i started training it , i used 24 reader but all what i see is

) -> (device: 0, name: GeForce GTX 980 Ti, pci bus id: 0000:23:00.0)
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/
12982 training images in /media/rmmal/data/East_data/

and the gpu utilization is 95% and all the 24 core are 100%

it has been 1 hour till now and nothing changed ?

so is there any thing wrong happened ?

[argman]

I dont know, can you provide a training sample or you can check you annotations, see samples in training_samples/

[rmmal]

it stuck alot in sleep(0.01)

I had a dataset that has x1,y1,x2,y2 and it's all about 1 class
so i converted it to
x1,y1,x2,y1,x2,y2,x1,y2 ,class1

for each image it may go through sleep(0.01) alot of times between reading the image in the generator function and the end of this function

what could be wrong ?

also i have a GPU that has 6GB ram would it be a problem ?

[argman]

you may try to reduce the number of readers and batch size

[InesBeltaief]

hello @rmmal can you please share the code that you used to convert annotations?

[yangxuehang]

@rmmal hello, how did you fix the problem about sleep(0.01) ?

[rmmal]

@InesBeltaief if you have x1 , y1 , x2 , y2
then you can write them in such format
file_write.write(x1+","+y1+","+x2+","+y1+","+x2+","+y2+","+x1+","+y2+",text\n")

[rmmal]

@yangxuehang there was a problem in my paths and dataset