minImages Training Variable

[camjac251]

I saw that by default dfaker will max out the used images for a given data set at 2000. My A set is under 2000 while my B set is around 12000. Is this code minImages = 2000 by design to only take in 2000 at a time and it randomizes them? When the script first loads it doesn't just load the first 2000 but all of the images, only the output afterward shows that it's maxing at 2000. Does the script choose the best 2000 to use?

If it doesn't, would it be possible to set it to use every photo from each data set? So for example, using 1500 from A and 6204 from B?
When I use the commented extra like this minImages = min(len(images_A),len(images_B))*20, it fails to run.

[camjac251]

When testing I had set minImages = 12000#min(len(images_A),len(images_B))*20 in my train.py file. However when trying to run training again, this is the output/error I get. Is 12000 not feasible for the memory usage that it would take up? I noticed that in my training with only 2000, when looking at the preview I don't see images near the end of the data set.

python train.py
running
loading C:/df/project1/data/A: 100%| 1635/1635 [00:06<00:00, 264.47it/s]
loading C:/df/project1/data/B: 100%| 11819/11819 [00:55<00:00, 212.27it/s]
Images A (1508, 256, 256, 4)
Images B (10332, 256, 256, 4)
Traceback (most recent call last):
  File "train.py", line 32, in <module>
    images_B = images_B/255.0
MemoryError

[ytn2001]

minImages = max(len(images_A),len(images_B))

[camjac251]

That gives back the same MemoryError.

[feepdakenoob]

You probably don't have enough memory to load ~12k images.

In my local I changed that to max(len(images_A),len(images_B)) just as ytn2001 did

And then I added a sanity check that if minImages is greater than 6000 log a warning and set it to 6000

[camjac251]

You use 6000 now? over 4000? Do you know if it's the VRAM or RAM that's the limitation. I'm guessing if you are using 6000 images, then you'd have 12GB of vram.

[dfaker]

This is a limitation around keeping all of the images in memory (RAM).