Reduce memory consumption in blob_* functions with preallocation #6597
Conversation
Initialization of image_cube reduce the max memory consumption by X% (depending on dims and num_sigma), which helped me run the code on weaker machine. With pre-allocating of the output array, we avoid using `np.stack` Testing: ``` from math import sqrt import skimage from skimage import data from skimage.feature import blob_dog, blob_log, blob_doh from skimage.color import rgb2gray import numpy as np import matplotlib.pyplot as plt image = data.hubble_deep_field()[:700, :700] image_gray = rgb2gray(image) Ds = 20 image_grayD = np.stack([image_gray for i in range(Ds)],2) print(image_grayD.shape) blobs_log = blob_log(image_grayD, max_sigma=40, min_sigma = 5, num_sigma=20, threshold=.01) ``` Max Ram: Without change - 3.247GB With change - 4.707GB
orena1
changed the title
|
Thank you @orena1 for working on these memory improvements. I took the liberty to change a few minor things and merged our main branch into this PR. The latter hopefully fixes the CI. |
lagru
changed the title
|
@jarrodmillman, @stefanv Do you see an immediate answer to why build fails on the linux CI? It doesn't seem to detect ninja although ninja 1.11.1 is present. |
I did not know the empty default type is float, it is more efficient to initialize it with the same dtype of the image
|
Hi @lagru, I added another change, I did not know that the default of empty is float64; the image can be float32 sometimes, so it is better to initialize empty with the correct datatype. |
|
Apologies for not seeing this before opening #6621. We had implemented the changes independently upstream in rapidsai/cucim#413 when porting the blob detectors there and i was backporting that here and did not properly check for an open PR first. Can you pull the Alternatively, I can also just add co-authored-by credit for you in the other PR. |
reduce memory consumption by preallocating image_cube
|
@lagru, please give a 2nd look when you get a chance. I think it is ready to merge |
to hide flake8 complaints about to long lines in those docstrings due to long URLs.
|
Great, thanks |
Description
preallocating of image_cube reduce the max memory consumption by 30% (depending on dims and
num_sigm), which helped me run the code on weaker machine. With pre-allocating of the output array, we avoid usingnp.stackTesting:
Max Ram:
Without change - 4.707GB
With change - 3.247GB