Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Memory leak #17

Closed
chetan opened this Issue · 8 comments

4 participants

@chetan

We observed hue/django using more than 3gb memory today -

hue       3649  0.0 24.1 3900596 3711988 ?     S    Dec12   0:41 /usr/bin/python2.6

It was up for only about 3 days. We're running via CDH 4.1.2 on CentOS 6:

hue-2.1.0+221-1.cdh4.1.2.p0.9.el6.x86_64

Not sure what other info I can provide. I'll update the ticket if we see it happening again.

@romainr
Owner

It is the first time we heard about this. What exact command are you using for displaying the memory usage?

Is django_debug_mode=0 in hue.ini?

@chetan

The output above is from ps auxw | grep django. Debug is disabled.

@jrohland

I'm a coworker of chetan. I checked the hue process today and saw that it was consuming 52.1 % of the RAM on the box (7.8 gigs).

hue        878  0.0 52.1 8199192 8010788 ?     S    Dec14   3:40 /usr/bin/python2.6 -c import sys; from spawning import spawning_child; spawning_child.main() 849 3 15 spawning.django_factory.config_factory {"app_factory": "spawning.django_factory.app_factory", "access_log_file": "/dev/null", "status_port": 0, "port": 8888, "verbose": null, "deadman_timeout": 1, "source_directories": ["/usr/share/hue/desktop/core/src/desktop"], "pidfile": null, "args": ["desktop.settings"], "max_age": null, "num_processes": 1, "watch": null, "host": "**************", "coverage": null, "ssl_private_key": null, "sysinfo": null, "status_host": "**************", "ssl_certificate": null, "argv_str": "--factory=spawning.django_factory.config_factory desktop.settings --port 8888 -s 1 -t 0", "no_keepalive": null, "reload": null, "django_settings_module": "desktop.settings", "threadpool_workers": 0}
@romainr
Owner

What is the typical usage of Hue? Any logs?

Are you using Beeswax and downloading the results directly? Opening file in JobBrowser?

@jrohland

I mostly use Hue for the HDFS file manager to browse and look at files. It seems that the memory usage jumps the most when just previewing files. I just tried viewing two separate 13mb text files through the file browser and memory usage jumped up to 2.4% (around 360mb) from .2% and didn't drop back down.

I've also been using Beeswax to query impala but that doesn't seem to impact the memory usage as much as just browsing files. I do use the job browser as well but haven't noticed if that is impacting memory consumption.

Let me know if there's specific log files you'd like me to upload.

@romainr romainr was assigned
@romainr
Owner

I uploaded a 500MB file and when I change the length GET parameter to 40096 instead of 4096 and press F5 for a bit I indeed see the memory going up and now decreasing after.

I created https://issues.cloudera.org/browse/HUE-981

@abec
Collaborator

I've updated https://issues.cloudera.org/browse/HUE-981.

Basically, spawning is not letting go of memory for requests that do not finish (IE: when a page is refreshed multiple times very quickly). This is likely do to usage... clicking next or previous block several times accidentally.

We may try to address the issue with Spawning in the future... until then, a quick solution is to use cherrypy server.

@romainr
Owner

Cherrypy is now the webserver in Hue 3.0

@romainr romainr closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.