This repository has been archived by the owner. It is now read-only.

Memory Leak: Phantomjs 2 checked out on 13th November 2014 #12903

Closed
surenderthakran opened this Issue Jan 25, 2015 · 48 comments

Comments

Projects
None yet
@surenderthakran

surenderthakran commented Jan 25, 2015

Phantomjs version: Phantomjs 2 (built from source on 13th Nov 14 from master)

I wrote a script to scan websites using phantomjs. I open not more than 4 simultaneous pages at any point.
Some serious memory leak happens when i run it

page.settings.loadImages = false;

I run my script on an amazon linux ec2 server with a 1GB ram and a single core processor (its one of the lighter servers).

When not loading images my ram usage gradually increases to 99% as seen by the "top" command before phantom crashes.
However, when loading images it stabilizes at around 6-7%.

@alexey89777

This comment has been minimized.

alexey89777 commented Jan 25, 2015

+1 have the same issue too

@ariya

This comment has been minimized.

Owner

ariya commented Jan 27, 2015

Can you build the tag 2.0.0 and try it again? If that still happens, I'll mark this as Regression.

@surenderthakran

This comment has been minimized.

surenderthakran commented Jan 27, 2015

@ariya ok will try again on 2.0.0

@iradul

This comment has been minimized.

Contributor

iradul commented Feb 3, 2015

This is WebKit related issue :
https://bugreports.qt.io/browse/QTBUG-34494 (https://codereview.qt-project.org/#/c/76934/)
https://bugreports.qt.io/browse/QTBUG-36530
https://bugreports.qt.io/browse/QTBUG-38857

Take a look at this test script, you'll see how memory usage grows up to 1GB with --load-images=false but with --load-images=true it doesn't grow more then 350MB:

var page = require('webpage').create(),
    u = ["http://www.google.com/","https://www.facebook.com/","https://www.youtube.com/","https://www.yahoo.com/","http://www.amazon.com/","http://www.bing.com/","http://www.ebay.com/","http://www.wikipedia.org/","http://detroit.craigslist.org/","https://www.linkedin.com/","https://login.live.com/login.srf?wa=wsignin1.0&rpsnv=12&ct=1422984414&rver=6.4.6456.0&wp=MBI_SSL_SHARED&wreply=https:%2F%2Fmail.live.com%2Fdefault.aspx%3Frru%3Dinbox&lc=1033&id=64855&mkt=en-us&cbcxt=mai","https://twitter.com/","https://accounts.google.com/ServiceLogin?service=blogger&passive=1209600&continue=https://www.blogger.com/home&followup=https://www.blogger.com/home&ltmpl=start","http://www.aol.com/","http://go.com/","https://www.pinterest.com/","http://www.msn.com/","https://www.tumblr.com/","http://www.cnn.com/","http://www.ask.com/","http://www.huffingtonpost.com/","https://www.netflix.com/us/","https://www.paypal.com/home","http://www.weather.com/","http://conduit.com/","http://espn.go.com/","http://instagram.com/","https://wordpress.com/","https://www.bankofamerica.com/","http://akamihd.net/","http://www.imdb.com/","https://www.chase.com/","http://www.microsoft.com/en-us/default.aspx","http://www.about.com/","http://www.avg.com/us-en/homepage","http://www.pornhub.com/","http://xfinity.comcast.net/tt2/?cid=BBI","http://www.foxnews.com/","http://www.apple.com/","http://www.walmart.com/","http://xhamster.com/","http://home.mywebsearch.com/index.jhtml","https://www.wellsfargo.com/","http://www.xvideos.com/","http://www.yelp.com/","http://imgur.com/","http://www.nytimes.com/","http://www.nbcnews.com/","http://www.cnet.com/","http://www.reddit.com/","http://www.adobe.com/","http://www.ehow.com/","http://www.pandora.com/","http://www.pch.com/","http://www.hulu.com/","http://www.zedo.com/","https://www.etsy.com/","https://www.flickr.com/","http://www.outbrain.com/","http://optmd.com/","http://www.indeed.com/","http://new.livejasmin.com/en/?showPreviousVersionLink=0","http://www.zillow.com/","http://www.target.com/","http://www.xnxx.com/","http://www.homedepot.com/","http://www.redtube.com/","http://www.answers.com/","http://www.att.com/","http://www.shopathome.com/","http://www.wikia.com/Wikia","http://www.dailymail.co.uk/ushome/index.html","https://www.usps.com/","http://www.babylon.com/","http://www.ups.com/","http://www.bestbuy.com/","http://www.youporn.com/","http://www.reference.com/","https://www.godaddy.com/","http://www.groupon.com/","http://www.deviantart.com/","http://www.usatoday.com/","http://www.pof.com/","https://www.capitalone.com/","http://www.bbc.co.uk/","http://www.washingtonpost.com/","http://www.match.com/","http://drudgereport.com/","http://mlb.mlb.com/home","http://www.tripadvisor.com/","http://www.pogo.com/","http://www.verizonwireless.com/","https://accounts.google.com/ServiceLogin?service=blogger&passive=1209600&continue=https://www.blogger.com/home&followup=https://www.blogger.com/home&ltmpl=start","http://www.buzzfeed.com/","http://doublepimp.com/Account/LogIn","http://inksr.com/","http://www.fedex.com/","http://inksdata.com/","http://www.aweber.com/","http://abcnews.go.com/","https://vimeo.com/","https://hootsuite.com/","http://bleacherreport.com/","http://www.lowes.com/","http://www.yellowpages.com/","https://www.americanexpress.com/","http://www.tube8.com/","https://www.salesforce.com/"]
    i = 0;
page.onError = function() {}
function open() {
    var url = u.shift();
    if (url) {
        page.open(url, function(x) { 
            console.log(i++ + ' : ' + url + ' : ' + x);
            setTimeout(open, 2000);
        });
    } else {
        phantom.exit();
    }
}
open();
@rubemz

This comment has been minimized.

rubemz commented Jun 13, 2015

+1 here, any thoughts on a fix for this this?

@hgati

This comment has been minimized.

hgati commented Jun 14, 2015

+1 me too.

@dhruvbhatia

This comment has been minimized.

Contributor

dhruvbhatia commented Jun 22, 2015

+1 same issue

@ottolu

This comment has been minimized.

ottolu commented Jun 28, 2015

+1 here

@behrangsa

This comment has been minimized.

behrangsa commented Jul 21, 2015

Any ETAs for a fix for this issue?

@druxax

This comment has been minimized.

druxax commented Aug 7, 2015

+1 please fix it

1 similar comment
@manning23

This comment has been minimized.

manning23 commented Sep 14, 2015

+1 please fix it

@behrangsa

This comment has been minimized.

behrangsa commented Sep 15, 2015

Guys, this is a QT bug. If you want it fixed, please vote here: https://bugreports.qt.io/browse/QTBUG-38857

@Tomtomgo

This comment has been minimized.

Contributor

Tomtomgo commented Sep 15, 2015

Voted.

@mepard

This comment has been minimized.

Contributor

mepard commented Sep 26, 2015

Here's a workaround that was adequate for my purposes: set loadImages to true, then use onResourceRequested to abort requests for image urls. In my case, I'm using casper.on('resource.requested'). It seems to stop the leaks without other side effects.

@brovikov

This comment has been minimized.

brovikov commented Oct 10, 2015

@behrangsa voted

@ZhiweiWang

This comment has been minimized.

ZhiweiWang commented Nov 10, 2015

@mepard good idea!

@maZahaca

This comment has been minimized.

maZahaca commented Nov 13, 2015

+1 to fix it

@maZahaca

This comment has been minimized.

maZahaca commented Nov 13, 2015

@mepard I'm getting error: TypeError: Object # has no method 'abort'
on attempt to abort request. PhantomJS 2.0.

@mepard

This comment has been minimized.

Contributor

mepard commented Nov 16, 2015

This is working for me (OS X 10.11 and Windows 10):

function ResourceRequested (request, networkRequest)
{
    if (options.workAroundLoadImagesLeak)
    {
        // Work around memory leak in WebKitQT in PhantomJS 2.0.0 when not loading images.
        // In initialize above we told PhantomJS to always load images, but here we'll suppress
        // the image resources.
        if (/\.(jpg|jpeg|png|gif|tif|tiff|mov)$/i.test(request.url))
        {
            if (options.trackResources)
            {
                console.log(moment().format() + ': Suppressing image #' + request.id + ': ' + request.url);
            }
            networkRequest.abort();
            return;
        }
    }
}
@mepard

This comment has been minimized.

Contributor

mepard commented Jan 20, 2016

FWIW, https://codereview.qt-project.org/#/c/76934/2 claims to have fixed this, but it doesn't seem like it to me. The master branch currently has the change, but the leak still happens.

@behrangsa

This comment has been minimized.

behrangsa commented Jan 21, 2016

mepard added a commit to mepard/qtwebkit that referenced this issue Jan 28, 2016

@mepard

This comment has been minimized.

Contributor

mepard commented Jan 28, 2016

Turns out it's a WebKit bug. I suspect it's the same as https://bugs.webkit.org/show_bug.cgi?id=17469 even though that report doesn't mention autoLoadImages.

@mepard

This comment has been minimized.

Contributor

mepard commented Feb 25, 2016

I just did a run where I visited around 330 pages with lots of images. Without this patch memory consumption hit around 2.5 GB, with the patch it was about 500 MB.

@coderzbx

This comment has been minimized.

coderzbx commented Apr 21, 2016

set option:load_image as True, resolve my problem

@tolark

This comment has been minimized.

tolark commented May 3, 2016

It has been 3 years without any fixes. 2.1.1 has still memory leak issue with python 2.7 in all operating systems.

@behrangsa

This comment has been minimized.

behrangsa commented May 8, 2016

I appreciate that this is an open source project, but if you were an end user of your own project and had to eat your own dog's food, would you be happy with such an important bug to be left unfixed for three years?

At least start a campaign, ask for an amount of donation that you think would be enough to fix this problem, ask people to contribute and donate money, and then focus on this issue and fix it.

Goddamit ;) https://youtu.be/0ubcetW9-D8?t=4s

@ovninosa

This comment has been minimized.

ovninosa commented May 9, 2016

+1 fix please!

@psellers89

This comment has been minimized.

psellers89 commented Jul 14, 2016

+1 ...

@volvofixthis

This comment has been minimized.

volvofixthis commented Jul 25, 2016

+1

2 similar comments
@Chort409

This comment has been minimized.

Chort409 commented Aug 1, 2016

+1

@onedesert

This comment has been minimized.

onedesert commented Aug 23, 2016

+1

@behrangsa

This comment has been minimized.

behrangsa commented Aug 24, 2016

...20 years later: +1

@maZahaca

This comment has been minimized.

maZahaca commented Aug 24, 2016

ahaha, yeah! old guys are still pressing the button "+1"

@psychok7

This comment has been minimized.

psychok7 commented Aug 30, 2016

+1

6 similar comments
@xmiao

This comment has been minimized.

xmiao commented Sep 22, 2016

+1

@max-favilli

This comment has been minimized.

max-favilli commented Oct 29, 2016

+1

@arkkanoid

This comment has been minimized.

arkkanoid commented Nov 7, 2016

+1

@timurengineer

This comment has been minimized.

timurengineer commented Nov 8, 2016

+1

@toldroyd

This comment has been minimized.

toldroyd commented Nov 8, 2016

+1

@dvygolov

This comment has been minimized.

dvygolov commented Nov 10, 2016

+1

@matthewstifler

This comment has been minimized.

matthewstifler commented Nov 12, 2016

+1 (please)

@anexplore

This comment has been minimized.

anexplore commented Dec 2, 2016

+1 please!!

@jorgeluisdelgado

This comment has been minimized.

jorgeluisdelgado commented Jan 10, 2017

+1 Please, this is still an issue 1 year later. :(

@grigoriy-didorenko

This comment has been minimized.

grigoriy-didorenko commented Jan 26, 2017

+1

@Vitallium

This comment has been minimized.

Collaborator

Vitallium commented Feb 7, 2017

Let's track this in #11390

@Vitallium Vitallium closed this Feb 7, 2017

@threematesarl

This comment has been minimized.

threematesarl commented May 3, 2017

+1

maxlazio pushed a commit to gitlabhq/gitlabhq that referenced this issue Jun 13, 2017

Let PhantomJS load local images
This change fix a memory leak due to a Webkit bug:
ariya/phantomjs#12903

Also:

- Whitelist only localhost and 127.0.0.1 in Capybara + JS specs
- Blacklist all requests to media such as images, videos, PDFs, CSVs etc.
- Log all the requests made.

Signed-off-by: Rémy Coutable <remy@rymai.me>
@ChasingRainbows

This comment has been minimized.

ChasingRainbows commented Sep 9, 2017

+1 (2.1.1)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.