Network training

[BavoKempen]

Dear Kevin and Russell,

I have been trying training a new network out of the image bouts that I created from our data. Unfortunately, an error occurred and thus I started training a new network for each file that we have. Consequently, I have singled out the problematic files that give the error below.

Step 1 of 4: Training a Region Proposal Network (RPN).
Error using visionBboxIntersectByUnion
Requested 844x3812727 (24.0GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may
take a long time and cause MATLAB to become unresponsive. See array size limit or preference panel for more information.

Error in bboxOverlapRatio (line 81)
        overlapRatio = visionBboxIntersectByUnion(bboxA, bboxB);

Error in vision.internal.cnn.rpn.selectTrainingSamples (line 50)
    iou = bboxOverlapRatio(groundTruth, regionProposals, 'union');

Error in vision.internal.cnn.rpn.RPNRegionReader/fillTrainingSamples (line 249)
                    s(i) = vision.internal.cnn.rpn.selectTrainingSamples(params, c{:});

Error in vision.internal.cnn.rpn.RPNRegionReader (line 158)
            this.TrainingSamples = fillTrainingSamples(this, groundTruth, params.UseParallel);

Error in vision.internal.cnn.rpn.RPNTrainingRegionDispatcher (line 64)
                vision.internal.cnn.rpn.RPNRegionReader(...

Error in fasterRCNNObjectDetector/createRPNTrainingDispatcher (line 665)
            dispatcher = vision.internal.cnn.rpn.RPNTrainingRegionDispatcher(...

Error in fasterRCNNObjectDetector.trainRPN (line 165)
            dispatcher = createRPNTrainingDispatcher(detector, groundTruth, opts, inputSize, params);

Error in trainFasterRCNNObjectDetector (line 289)
    [stage1Detector, rpn, info] = fasterRCNNObjectDetector.trainRPN(trainingData, rpn, options(1),  iStageOneParams(params),
    checkpointSaver);

Error in TrainSqueakDetector (line 61)
detector = trainFasterRCNNObjectDetector(TrainingTables, layers, options, ...

Error in trainnew_Callback (line 41)
        [detector, layers, options] = TrainSqueakDetector(TrainingTables);

Error in gui_mainfcn (line 95)
        feval(varargin{:});

Error in DeepSqueak (line 30)
    gui_mainfcn(gui_State, varargin{:});

Error in
matlab.graphics.internal.figfile.FigFile/read>@(hObject,eventdata)DeepSqueak('trainnew_Callback',hObject,eventdata,guidata(hObject)) 
Error while evaluating Menu Callback

Furthermore, I have compared the files that create this error with similar files to check for size. However, there are other files containing more USVs and are longer in time than the ones creating the above mentioned error. Also, running it on the stronger computer gave the same error.
Is there a way around this to reduce the array sizes of these files or does the problem lie somewhere else?

Kind regards,

Bavo Kempen

[DrCoffey]

Hey Bavo,

Sorry it has taken us a while to get around to these issues, it has been a busy couple weeks. It looks like you may just have a single enormous image somewhere in the training data. How big are the bouts you are training with? We generally use 1 second.

-Kevin

[BavoKempen]

Hey Kevin,

I have checked and something peculiar occured, for some reason it transformed a file of 5 minutes into 4 images where the largest file is 9.57Mb and has the following measurements 61280 x 275. Ofcourse this is impossible that I only got 4 images for this file, since there should be 300 more or less. However, I did not change the default settings of creating bouts aside from increasing the augmented duplicates. However, I just retried with default 1 augmented duplicate and I get the same result.

Any idea why this might occur and I do not get the expected 300 images, also, I checked my other files that do work during training and they are not an integer multiple of their lenght in seconds.

I hope this helps!

Kind regards,

Bavo

[DrCoffey]

Hey Bavo,

That is peculiar... Can you send me the audio file and the detection file to take a look at?

-Kevin

[DrCoffey]

Hey Bavo,

The way bouts works on the default settings (1s) is that a new image will be created when there are no calls for over 1s. That animal just never stops making calls for more than a 1 second! If you reduce the bout size to 0.25s then it will produce a bunch of normal sized images that will work for training.

-Kevin

[BavoKempen]

Hey Kevin,

Everything works fine now, thus, if you'd like you can close this issue :)
Thanks again for all the help.
Kind regards,

Bavo

[DrCoffey]

That's great, feel free to open another if you run into anything new.