Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resizing images causes memory leaks #1443

Closed
ducanh997 opened this issue May 13, 2022 · 9 comments · May be fixed by #1623
Closed

Resizing images causes memory leaks #1443

ducanh997 opened this issue May 13, 2022 · 9 comments · May be fixed by #1623
Assignees
Labels

Comments

@ducanh997
Copy link

ducanh997 commented May 13, 2022

Thumbor request URL

http://localhost:8888/unsafe/30000x30000/https://raw.githubusercontent.com/thumbor/thumbor/master/example.jpg

Expected behaviour

Calling the request URL above multiple times shouldn't cause a memory leak

Actual behaviour

When I call the above request URL multiple times, my server's memory keeps increasing and results in an OOM error (Process finished with exit code 137, interrupted by signal 9: SIGKILL). I found a workaround to solve this problem is to add gc.collect() at the _cleanup() method, but doing this results in much higher CPU usage in return.

Operating system

Ubuntu 20.04.4 LTS

Your thumbor.conf

# -*- coding: utf-8 -*-
# thumbor imaging service
# https://github.com/thumbor/thumbor/wiki

# Licensed under the MIT license:
# http://www.opensource.org/licenses/mit-license
# Copyright (c) 2011 globo.com thumbor@googlegroups.com

from os.path import expanduser, join

home = expanduser("~")

## the domains that can have their images resized
## use an empty list for allow all sources
# ALLOWED_SOURCES = ["mydomain.com"]

## the max width of the resized image
## use 0 for no max width
## if the original image is larger than MAX_WIDTH x MAX_HEIGHT,
## it is proportionally resized to MAX_WIDTH x MAX_HEIGHT
# MAX_WIDTH = 800

## the max height of the resized image
## use 0 for no max height
## if the original image is larger than MAX_WIDTH x MAX_HEIGHT,
## it is proportionally resized to MAX_WIDTH x MAX_HEIGHT
# MAX_HEIGHT = 600

## the quality of the generated image
## this option can vary widely between
## imaging engines and works only on jpeg images
## QUALITY = 85
# WEBP_QUALITY = 80

## Automatically converts images to WebP if Accepts header present
# AUTO_WEBP = False

## Automatically converts PNG to JPG if PNG has no transparency.
## WARNING: Depending on case, this is not a good deal. This
## transformation maybe causes distortions or the size of image can increase.
## Images with texts, for example, the result image maybe will be distorced.
## Dark images, for example, the size of result image maybe will be bigger.
## You have to evaluate the majority of your use cases to take a decision about
## this conf.
# AUTO_PNG_TO_JPG = False

## Specify the ratio between 1in and 1px for SVG images. This is only used when
## rasterizing SVG images having their size units in cm or inches.
# SVG_DPI = 150

## Preserves exif information in generated images. Increases image size in
## kbytes, use with caution.
PRESERVE_EXIF_INFO = False

## enable this options to specify client-side cache in seconds
MAX_AGE = 24 * 60 * 60

## client-side caching time for temporary images (using queued detectors or
## after detection errors)
MAX_AGE_TEMP_IMAGE = 0

## Sends If-Modified-Since & Last-Modified headers; requires support from result
## storage
SEND_IF_MODIFIED_LAST_MODIFIED_HEADERS = False

## Sends the Access-Control-Allow-Origin header
# ACCESS_CONTROL_ALLOW_ORIGIN_HEADER = "*"

## the way images are to be loaded
LOADER = "thumbor.loaders.http_loader"
# LOADER = "thumbor.loaders.file_loader"
# LOADER = "thumbor.compatibility.loader"

## the path to find images
# FILE_LOADER_ROOT_PATH = "./tests/fixtures/images"

## maximum size of the source image in Kbytes.
## use 0 for no limit.
## this is a very important measure to disencourage very
## large source images.
## THIS ONLY WORKS WITH http_loader.
MAX_SOURCE_SIZE = 0

## if you set UPLOAD_ENABLED to True,
## a route /upload will be enabled for your thumbor process
## You can then do a put to this URL to store the photo
## using the specified Storage
UPLOAD_ENABLED = False
UPLOAD_PHOTO_STORAGE = "thumbor.storages.file_storage"
UPLOAD_PUT_ALLOWED = False
UPLOAD_DELETE_ALLOWED = False
UPLOAD_DEFAULT_FILENAME = "image"

## how to store the loaded images so we don't have to load
## them again with the loader
# STORAGE = "thumbor.storages.redis_storage"
# STORAGE = "thumbor.storages.no_storage"
STORAGE = "thumbor.storages.file_storage"
# STORAGE = "thumbor.storages.mixed_storage"
# STORAGE = "thumbor.compatibility.storage"

## root path of the file storage
FILE_STORAGE_ROOT_PATH = join(home, "thumbor", "storage")

## If you want to cache results, use this options to specify how to cache it
## Set Expiration seconds to ZERO if you want them not to expire.
RESULT_STORAGE = "thumbor.result_storages.file_storage"
# RESULT_STORAGE = "thumbor.compatibility.result_storage"
RESULT_STORAGE_EXPIRATION_SECONDS = 60 * 60 * 24  # one day
RESULT_STORAGE_FILE_STORAGE_ROOT_PATH = join(home, "thumbor", "result_storage")

RESULT_STORAGE_STORES_UNSAFE = False

## stores the crypto key in each image in the storage
## this is VERY useful to allow changing the security key
STORES_CRYPTO_KEY_FOR_EACH_IMAGE = True

## imaging engine to use to process images

ENGINE = "thumbor.engines.pil"

## if you need to use the OpenCV engine please refer
## to https://github.com/thumbor/opencv-engine
# ENGINE = "opencv_engine"

## if you need to use the GraphicsMagick engine please refer
## to https://github.com/thumbor/graphicsmagick-engine
# ENGINE = "graphicsmagick_engine"


## The graphics engine (and any optimizers) will block the IOLoop.  If you want
## to increase the number of concurrent requests Thumbor can handle, increase
## ENGINE_THREADPOOLSIZE to use a thread pool.  Benchmarks indicate 1xnumber of
## cpus - 2x number of cpus gives the best results, but do some benchmarks with
## your workload.
# ENGINE_THREADPOOL_SIZE = 0

## detectors to use to find Focal Points in the image
## more about detectors can be found in thumbor's docs
## at https://github.com/thumbor/thumbor/wiki
DETECTORS = [
    #"thumbor.detectors.face_detector",
    #"thumbor.detectors.profile_detector",
    #"thumbor.detectors.glasses_detector",
    #"thumbor.detectors.feature_detector",
]

## Redis parameters for queued detectors
# REDIS_QUEUE_SERVER_HOST = "localhost"
# REDIS_QUEUE_SERVER_PORT = 6379
# REDIS_QUEUE_SERVER_DB = 0
# REDIS_QUEUE_SERVER_PASSWORD = None

## AWS SQS parameters for queued detectors
# SQS_QUEUE_KEY_ID = ""
# SQS_QUEUE_KEY_SECRET = ""
# SQS_QUEUE_REGION = "us-east-1"

## if you use face detection this is the file that
## OpenCV will use to find faces. The default should be
## fine, so change this at your own peril.
## if you set a relative path it will be relative to
## the thumbor/detectors/face_detector folder
# FACE_DETECTOR_CASCADE_FILE = "haarcascade_frontalface_alt.xml"

## this is the security key used to encrypt/decrypt urls.
## make sure this is unique and not well-known
## This can be any string of up to 16 characters
SECURITY_KEY = "MY_SECURE_KEY"

## if you enable this, the unencryted URL will be available
## to users.
## IT IS VERY ADVISED TO SET THIS TO False TO STOP OVERLOADING
## OF THE SERVER FROM MALICIOUS USERS
ALLOW_UNSAFE_URL = True

## Mixed storage classes. Change them to the fullname of the
## storage you desire for each operation.
# MIXED_STORAGE_FILE_STORAGE = "thumbor.storages.file_storage"
# MIXED_STORAGE_CRYPTO_STORAGE = "thumbor.storages.no_storage"
# MIXED_STORAGE_DETECTOR_STORAGE = "thumbor.storages.no_storage"


## This configuration indicates whether thumbor should use a
## custom error handler.
## Defaults to: False
# USE_CUSTOM_ERROR_HANDLING = False
# ERROR_HANDLER_MODULE = "thumbor.error_handlers.sentry"
# SENTRY_DSN_URL = ""
# SENTRY_ENVIRONMENT = None

# from thumbor.filters import BUILTIN_FILTERS
# FILTERS = BUILTIN_FILTERS

## can only be applied if there are already points for the image being served
## this means that either you are using the local face detector or the image
## has already went through remote detection
# FILTERS = BUILTIN_FILTERS + [
#    "thumbor.filters.redeye",
# ]

# OPTIMIZERS = [
#    "thumbor.optimizers.jpegtran",
#    "thumbor_plugins.optimizers.gifv",
# ]

# from thumbor.handler_lists import BUILTIN_HANDLERS

# HANDLER_LISTS = BUILTIN_HANDLERS + [
#   # Add your handler list modules here
# ]

# JPEGTRAN_PATH = "/usr/local/bin/jpegtran"

## If the CurlAsyncHTTPClient should be used
## Defaults to: False
HTTP_LOADER_CURL_ASYNC_HTTP_CLIENT = False

## If the CurlAsyncHTTPClient is being used and
## should timeout on slow download speed.
## SPEED in bytes/s and TIME in s
## Dfaults to: 0 (no timeout)
# HTTP_LOADER_CURL_LOW_SPEED_TIME = 0
# HTTP_LOADER_CURL_LOW_SPEED_LIMIT = 0

################################# Compatibility ################################

## Loader that will be used with the compatibility layer, instead of the
## compatibility loader. Please only use this if you can't use up-to-date
## loaders.
## Defaults to: None
# COMPATIBILITY_LEGACY_LOADER = None

## Storage that will be used with the compatibility layer, instead of the
## compatibility storage. Please only use this if you can't use up-to-date
## storages.
## Defaults to: None
# COMPATIBILITY_LEGACY_STORAGE = None

## Result Storage that will be used with the compatibility layer, instead of
## the compatibility result storage. Please only use this if you can't use
## up-to-date result storages.
## Defaults to: None
# COMPATIBILITY_LEGACY_RESULT_STORAGE = None

################################################################################

I'm using default thumbor.conf. I tried changing some configs like ENGINE_THREADPOOL_SIZE or GC_INTERVAL, but it doesn't solve the problem.

@ducanh997 ducanh997 changed the title Resizing images causes memory leak errors Resizing images causes memory leaks May 13, 2022
@mvhirsch
Copy link
Contributor

Thanks for reaching out. Could you please post your thumbor.conf and your version of thumbor you're using, to provide a minimum reproduceable build?

@ducanh997
Copy link
Author

ducanh997 commented Jun 13, 2022

@mvhirsch I updated my thumbor.conf in the description above. I'm using thumbor 7.0.10. Thanks for your support.

@github-actions
Copy link

This issue is stale because it has been open 30 days with no activity. Remove the stale label or add a comment, or this issue will be closed in 5 days. You can always re-open if you still feel this is still an issue. Tag @heynemann for more information.

@github-actions github-actions bot added the Stale label Jul 13, 2022
@github-actions
Copy link

This issue was closed because it has been stale for 5 days with no activity.

@RaphaelVRossi RaphaelVRossi self-assigned this Nov 16, 2023
@RaphaelVRossi
Copy link
Member

I'll open a PR to fix this. 🚀

Copy link

This issue is stale because it has been open 30 days with no activity. Remove the stale label or add a comment, or this issue will be closed in 15 days. You can always re-open if you still feel this is still an issue. Tag @heynemann for more information.

Copy link

This issue is stale because it has been open 30 days with no activity. Remove the stale label or add a comment, or this issue will be closed in 15 days. You can always re-open if you still feel this is still an issue. Tag @heynemann for more information.

Copy link

This issue is stale because it has been open 30 days with no activity. Remove the stale label or add a comment, or this issue will be closed in 15 days. You can always re-open if you still feel this is still an issue. Tag @heynemann for more information.

@github-actions github-actions bot added the Stale label Feb 16, 2024
Copy link

github-actions bot commented Apr 1, 2024

This issue was closed because it has been stale for 15 days with no activity.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Apr 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants