Classifier dies when processing a larger data set

[kamilzyla]

When running the classifier on a "small sample of data" provided by Robbie (628 images), the process runs quite slowly and dies on the second batch. I was using my work laptop with 16 GB of RAM. @marekrogala suggests the images might have unnecessarily large resolution and scaling them down could possibly help.

[kamilzyla]

We tried with @swiezew to reproduce the issue on his Mac, but the classification finished without issues. I need to investigate the issue on my laptop further.

[marekrogala]

@swiezew how long does it take on your machine? if we extrapolate to numbers of images that we expect in real usage, how long would the inference take? is it using GPU on your machine?

[swiezew]

@marekrogala I've worked with the 628 images sample. I used my 2013 Mac's CPU (2,7GHz, 16GB RAM). Inference was divided into 10 batches, each took approximately 100 seconds (15 minutes in total). That means each image took around 0.7sec. The typical size of the data to be analysed has been said be 30000 images, which would take around 6hours on my laptop.

[kamilzyla]

It seems the crash is indeed due to the classifier running out of memory. I noticed that I am consistently able to process the full data set if I close all Chrome instances; with a few tabs open the classifier would crash on the 10th batch and with many tabs open it would crash on the 3rd.

Our users probably won't have more than 16 GB RAM on their machines (?), so solving this issue is a priority. Scaling the images down might suffice; I'll create a separate issue for it.

[marekrogala]

@kamil Żyła <kamil@appsilon.com> one way to quickly test this is to use a different num_workers param, e.g. 4, or 2 and see what happens. IIRC by default it's 16 or similar. Does lower number of concurrent workers help?" Marek * Marek Rogala* CTO +48 509 368 092 | @marekrog <http://twitter.com/marekrog> czw., 9 lip 2020 o 08:13 Kamil Zyla <notifications@github.com> napisał(a):

…

It seems the crash is indeed due to the classifier running out of memory. I noticed that I am consistently able to process the full data set if I close all Chrome instances; with a few tabs open the classifier would crash on the 10th batch and with many tabs open it would crash on the 3rd. Our users probably won't have more than 16 GB RAM on their machines (?), so solving this issue is a priority. Scaling the images down might suffice; I'll create a separate issue for it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#74 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAK3BP2VMCB6CLYUKGMWIUTR2VNYZANCNFSM4OSXZNAA> .

[kamilzyla]

Closing - we didn't have any reports of this issue recently.