Malloc - Runs Out Of Memory

[IWriteThings]

• face_recognition version: latest
• Python version: 3.6.3
• Operating System: Arch Linux

Description

When attempting to identify faces, I use face_location with model="cnn". It should work but it instead fails.

What I Did

face_location = face_recognition.face_locations(image, model="cnn")

Traceback (most recent call last):
  File "example.py", line 12, in <module>
    compare_faces(unknown_persons,known_list)
  File "/home/aaron/Development/bulk-image-facial-recognition/ml_face.py", line 36, in compare_faces
    recognize_faces(image)
  File "/home/aaron/Development/bulk-image-facial-recognition/ml_face.py", line 19, in recognize_faces
    face_locations = face_recognition.face_locations(image, model="cnn")
  File "/usr/lib/python3.6/site-packages/face_recognition/api.py", line 111, in face_locations
    return [_trim_css_to_bounds(_rect_to_css(face.rect), img.shape) for face in _raw_face_locations(img, number_of_times_to_upsample, "cnn")]
  File "/usr/lib/python3.6/site-packages/face_recognition/api.py", line 95, in _raw_face_locations
    return cnn_face_detector(img, number_of_times_to_upsample)
RuntimeError: Error while calling cudaMalloc(&data, new_size*sizeof(float)) in file /home/aaron/Development/tmp/dlib/dlib/dnn/gpu_data.cpp:195. code: 2, reason: out of memory

[ageitgey]

How big is your image and how much memory does your GPU have onboard? It sounds like the GPU itself doesn't have enough memory. You can check with the nvidia-smi command line tool.

[IWriteThings]

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.98                 Driver Version: 384.98                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GPU         Off  | 00000000:01:00.0  On |                  N/A |
| N/A   46C    P8    N/A /  N/A |     34MiB /  1997MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       337      G   /usr/lib/xorg-server/Xorg                     32MiB |
+-----------------------------------------------------------------------------+

Images are approximately 800K

Is my GPU not good enough?

[IWriteThings]

Updating to the latest DLIB now gives the error -

Traceback (most recent call last):
  File "example.py", line 12, in <module>
    compare_faces(unknown_persons,known_list)
  File "/home/aaron/Development/bulk-image-facial-recognition/ml_face.py", line 36, in compare_faces
    recognize_faces(image)
  File "/home/aaron/Development/bulk-image-facial-recognition/ml_face.py", line 19, in recognize_faces
    face_locations = face_recognition.face_locations(image, model="cnn")
  File "/usr/lib/python3.6/site-packages/face_recognition/api.py", line 111, in face_locations
    return [_trim_css_to_bounds(_rect_to_css(face.rect), img.shape) for face in _raw_face_locations(img, number_of_times_to_upsample, "cnn")]
  File "/usr/lib/python3.6/site-packages/face_recognition/api.py", line 95, in _raw_face_locations
    return cnn_face_detector(img, number_of_times_to_upsample)
MemoryError

[ageitgey]

What is the size of the image in pixels? Does it work if you scale the image way down in size? Two gigs isn't very much for a GPU.

[IWriteThings]

Scaling the images down to 800x600 makes the application function. It would appear that my hardware is unable to handle images that are much larger than that. Thank you for your assistance in this matter.

[AS630275035]

Hi, Have u settled it? I'm now having the same question. how to do it??

[AS630275035]

it is about the batch(128), i changed it to 32. and it is well done

[Mekhak]

Hi All,

So what was the actual solution? How to fix the error without scaling the image down?

Thanks

[IWriteThings]

@Mekhak Your only solution is to get more ram and a better video card or to scale your images to a lower resolution.

[Mekhak]

Hi All,

@IWriteThings Down-scaling image solves the problem but it decreases the detection accuracy.

I am using 8GB Geforce 1070 TI.

But I am wondering how much GPU memory requires DLib's face_detection model at python face_locations(img, number_of_times_to_upsample) function call point? I have debugged the DLib's face detection part: the cudaMalloc out of memory crash happens here:

file: "dlib/tools/python/src/cnn_face_detector.cpp"
function: "cnn_face_detection_model_v1::detect(py::array, const int)"
line: 54: "auto dets = net(image);"

An high resolution image (say 1200:1340) with number_of_times_to_upsample = 2 is "eating" the whole 8 GB GPU memory and the cudaMalloc out of memory crash happens.

Can please anyone point whether the specified resolution, upsample number and memory usage are normal for face_detector model?

Thanks and Regards,
Mekhak

[moguzozcan]

@ageitgey @Mekhak Any update on this issue, I am also facing the same issue while I am calling face_locations() method in a loop. I have 4GB GPU memory.

[Mekhak]

@moguzozcan There is not a solution which can make face_locations() function not to use such a huge memory. The only things you can so are to use more powerful GPU and scale the input images down.

[YanngLiu]

My experience is 4GB-memory GTX 1050 Ti can handle 30002000 image, but failed with 32502170 one.
Also the GPU crashed and led system crash after the out-of-memory, but I didn't encounter this after reboot and retry.

[Abhi744]

I have GPU GeForce GTX 1080 Ti with 11177MiB.
while I try to run the CNN model with number_of_times_to_upsample> 2, I get the same error.
Also, the default image size of my data is 1920x1080 which I reduced to 1422x800 using :

def image_resize(image, width = None, height = None, inter = cv2.INTER_AREA):
    # initialize the dimensions of the image to be resized and  grab the image size
    dim = None
    (h, w) = image.shape[:2]

    # if both the width and height are None, then return the original image
    if the width is None and height is None:
        return image

    # check to see if the width is None
    if the width is None:
        # calculate the ratio of the height and construct the dimensions
        r = height / float(h)
        dim = (int(w * r), height)

    # otherwise, the height is None
    else:
        # calculate the ratio of the width and construct the dimensions
        r = width / float(w)
        dim = (width, int(h * r))

    # resize the image
    resized = cv2.resize(image, dim, interpolation = inter)

    # return the resized image
    return resized

How much further should I resize the image and how, to run it on a greater number of upsamples?

[ageitgey]

If you use number_of_times_to_upsample=3, you are asking it to double the image, double it again, and double it again. With a 1920x1080 image, that's a huge amount of pixel data, so it makes sense it won't fit into GPU memory.

On the other hand, there's no reason to shrink the image more (losing quality) only to turn around and upsample it more. That's not accomplishing anything except shrinking and un-shrinking the size of the image. It would be better to keep the image the original size and upsample less.

[isConic]

I'm running into the same issue.
I have the following dimension for the box that I'm searching in.

w=947, h=1109,

All my other cuda accelerated DL models are not throwing the same error.

[Bah1996]

I have memory problems too

1. corrupted size vs. prev_size while consolidating
2. malloc(): invalid next size (unsorted)
3. free(): corrupted unsorted chunks
   What is the reason for these errors?