Memory leak #17

Closed
chetan opened this Issue Dec 14, 2012 · 8 comments

Comments

Projects
None yet
4 participants

chetan commented Dec 14, 2012

We observed hue/django using more than 3gb memory today -

hue       3649  0.0 24.1 3900596 3711988 ?     S    Dec12   0:41 /usr/bin/python2.6

It was up for only about 3 days. We're running via CDH 4.1.2 on CentOS 6:

hue-2.1.0+221-1.cdh4.1.2.p0.9.el6.x86_64

Not sure what other info I can provide. I'll update the ticket if we see it happening again.

Member

romainr commented Dec 15, 2012

It is the first time we heard about this. What exact command are you using for displaying the memory usage?

Is django_debug_mode=0 in hue.ini?

chetan commented Dec 15, 2012

The output above is from ps auxw | grep django. Debug is disabled.

I'm a coworker of chetan. I checked the hue process today and saw that it was consuming 52.1 % of the RAM on the box (7.8 gigs).

hue        878  0.0 52.1 8199192 8010788 ?     S    Dec14   3:40 /usr/bin/python2.6 -c import sys; from spawning import spawning_child; spawning_child.main() 849 3 15 spawning.django_factory.config_factory {"app_factory": "spawning.django_factory.app_factory", "access_log_file": "/dev/null", "status_port": 0, "port": 8888, "verbose": null, "deadman_timeout": 1, "source_directories": ["/usr/share/hue/desktop/core/src/desktop"], "pidfile": null, "args": ["desktop.settings"], "max_age": null, "num_processes": 1, "watch": null, "host": "**************", "coverage": null, "ssl_private_key": null, "sysinfo": null, "status_host": "**************", "ssl_certificate": null, "argv_str": "--factory=spawning.django_factory.config_factory desktop.settings --port 8888 -s 1 -t 0", "no_keepalive": null, "reload": null, "django_settings_module": "desktop.settings", "threadpool_workers": 0}
Member

romainr commented Dec 19, 2012

What is the typical usage of Hue? Any logs?

Are you using Beeswax and downloading the results directly? Opening file in JobBrowser?

I mostly use Hue for the HDFS file manager to browse and look at files. It seems that the memory usage jumps the most when just previewing files. I just tried viewing two separate 13mb text files through the file browser and memory usage jumped up to 2.4% (around 360mb) from .2% and didn't drop back down.

I've also been using Beeswax to query impala but that doesn't seem to impact the memory usage as much as just browsing files. I do use the job browser as well but haven't noticed if that is impacting memory consumption.

Let me know if there's specific log files you'd like me to upload.

romainr was assigned Dec 20, 2012

Member

romainr commented Dec 20, 2012

I uploaded a 500MB file and when I change the length GET parameter to 40096 instead of 4096 and press F5 for a bit I indeed see the memory going up and now decreasing after.

I created https://issues.cloudera.org/browse/HUE-981

Collaborator

abec commented Jan 18, 2013

I've updated https://issues.cloudera.org/browse/HUE-981.

Basically, spawning is not letting go of memory for requests that do not finish (IE: when a page is refreshed multiple times very quickly). This is likely do to usage... clicking next or previous block several times accidentally.

We may try to address the issue with Spawning in the future... until then, a quick solution is to use cherrypy server.

Member

romainr commented Sep 6, 2013

Cherrypy is now the webserver in Hue 3.0

romainr closed this Sep 6, 2013

@jfota jfota pushed a commit to mapr/hue that referenced this issue Nov 17, 2015

IVarha Merge pull request #17 from mapr/mapr-20440
[MAPR-20440][Hue 3.9] AttributeError at /oozie/submit_workflow/1 'mod…
87b6759
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment