Allowed memory size #151

Open
mrahmadt opened this Issue Jul 21, 2012 · 8 comments

Projects

None yet

2 participants

@mrahmadt

Hi

FYI, if I run get_statusesUser_timeline in a loop to get around 500 timeline, most of the time I will face this memory issue

Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 630691 bytes) in EpiTwitter.php on line 224

Thanks

Owner
jmathai commented Jul 23, 2012

What's the code look like? Can you break it up into smaller chunks?

This is the code, I can see this issue when I try to process hundreds of twitter accounts (getting time lines)

$sql = 'SELECT * FROM reports';
$sth = $dblink->prepare($sql);
$sth->execute();
$reports = $sth->fetchAll(PDO::FETCH_ASSOC);

foreach($reports as $report){
foreach($report['followers'] as $follower){
print getTimeline($follower);
}
}

function getTimeline($twitter_id){
$options = array('user_id'=>null,'count'=>70,'include_entities'=>0,'trim_user'=>0,'exclude_replies'=>0,'include_rts'=>1,'contributor_details'=>0);
$options['user_id'] = $twitter_id;
print "Follower Timeline $twitter_id\n";
$twitterObj = bootTwitter($twitter_id);
try {
$rep = $twitterObj->get_statusesUser_timeline($options);
if($rep->code!=200){echo "code error\n";return false;}
} catch (Exception $e) {
return false;
}
}

function bootTwitter($twitter_id=0){
global $twitterObj,$OAuth,$dblink;

    $OAuth = getOAuth($twitter_id); //nothing just return array from database

    $twitterObj = TwitterOAuth($OAuth['token'],$OAuth['secret'],$OAuth['consumer_key'],$OAuth['consumer_key_secret']);

    //Check if we have error with this key
    try {
        $resp = $twitterObj->get('/account/verify_credentials.json');  
    } catch (Exception $e) {
    }
return $twitterObj;

}

function TwitterOAuth($oauth_token,$oauth_token_secret,$consumer_key=null,$consumer_secret=null){
if($consumer_key==null){
$consumer_key = TWITTER_CONSUMER_KEY;
$consumer_secret = TWITTER_CONSUMER_SECRET;
}
$twitterObj = new EpiTwitter($consumer_key, $consumer_secret);
$twitterObj->setToken($oauth_token, $oauth_token_secret);
return $twitterObj;
}

Owner
jmathai commented Jul 30, 2012

How many times is getTimeline called?

On Mon, Jul 30, 2012 at 3:32 AM, Ahmad AlTwaijiry <
reply@reply.github.com

wrote:

This is the code, I can see this issue when I try to process hundreds of
twitter accounts (getting time lines)

$sql = 'SELECT * FROM reports';
$sth = $dblink->prepare($sql);
$sth->execute();
$reports = $sth->fetchAll(PDO::FETCH_ASSOC);

foreach($reports as $report){
foreach($report['followers'] as $follower){
print getTimeline($follower);
}
}

function getTimeline($twitter_id){
$options =
array('user_id'=>null,'count'=>70,'include_entities'=>0,'trim_user'=>0,'exclude_replies'=>0,'include_rts'=>1,'contributor_details'=>0);
$options['user_id'] = $twitter_id;
print "Follower Timeline $twitter_id\n";
$twitterObj = bootTwitter($twitter_id);
try {
$rep = $twitterObj->get_statusesUser_timeline($options);
if($rep->code!=200){echo "code error\n";return false;}
} catch (Exception $e) {
return false;
}
}

function bootTwitter($twitter_id=0){
global $twitterObj,$OAuth,$dblink;

            $OAuth = getOAuth($twitter_id); //nothing just return

array from database

            $twitterObj =

TwitterOAuth($OAuth['token'],$OAuth['secret'],$OAuth['consumer_key'],$OAuth['consumer_key_secret']);

            //Check if we have error with this key
            try {
                    $resp =

$twitterObj->get('/account/verify_credentials.json');
} catch (Exception $e) {
}
return $twitterObj;
}

function
TwitterOAuth($oauth_token,$oauth_token_secret,$consumer_key=null,$consumer_secret=null){
if($consumer_key==null){
$consumer_key = TWITTER_CONSUMER_KEY;
$consumer_secret = TWITTER_CONSUMER_SECRET;
}
$twitterObj = new EpiTwitter($consumer_key, $consumer_secret);
$twitterObj->setToken($oauth_token, $oauth_token_secret);
return $twitterObj;
}


Reply to this email directly or view it on GitHub:
#151 (comment)

Around 80 times then it will fail.

Adding memory_get_usage to the loop "foreach($report['followers'] as $follower){"

print 'f memory_get_usage: ' . formatBytes(memory_get_usage()) . "\n";

f memory_get_usage: 995.32 KB
f memory_get_usage: 1.12 MB
f memory_get_usage: 1.31 MB
f memory_get_usage: 1.32 MB
f memory_get_usage: 1.51 MB
f memory_get_usage: 1.75 MB
f memory_get_usage: 1.8 MB
f memory_get_usage: 1.98 MB
f memory_get_usage: 1.99 MB
f memory_get_usage: 2.03 MB
f memory_get_usage: 2.16 MB
f memory_get_usage: 2.27 MB
f memory_get_usage: 2.54 MB
f memory_get_usage: 2.86 MB
f memory_get_usage: 2.86 MB
f memory_get_usage: 3.05 MB
f memory_get_usage: 3.11 MB
f memory_get_usage: 3.16 MB
f memory_get_usage: 3.16 MB
f memory_get_usage: 3.44 MB
f memory_get_usage: 3.44 MB
f memory_get_usage: 3.64 MB
f memory_get_usage: 3.67 MB
f memory_get_usage: 3.68 MB
f memory_get_usage: 3.82 MB
f memory_get_usage: 4.15 MB
f memory_get_usage: 4.18 MB
f memory_get_usage: 4.41 MB
f memory_get_usage: 4.54 MB
f memory_get_usage: 4.67 MB
f memory_get_usage: 4.81 MB
f memory_get_usage: 4.81 MB
f memory_get_usage: 4.95 MB
f memory_get_usage: 5.31 MB
f memory_get_usage: 5.62 MB
f memory_get_usage: 5.94 MB
f memory_get_usage: 6.16 MB
f memory_get_usage: 6.17 MB
f memory_get_usage: 6.33 MB
f memory_get_usage: 6.68 MB
f memory_get_usage: 6.76 MB
f memory_get_usage: 6.91 MB
f memory_get_usage: 6.92 MB
...
...
...

Hi

Just tried abraham/twitteroauth with the same code and everything is working without memory leak

f memory_get_usage: 1.31 MB
f memory_get_usage: 1.31 MB
f memory_get_usage: 1.31 MB
f memory_get_usage: 1.31 MB
f memory_get_usage: 1.31 MB
f memory_get_usage: 1.31 MB
f memory_get_usage: 1.31 MB
f memory_get_usage: 1.31 MB
f memory_get_usage: 1.31 MB
f memory_get_usage: 1.31 MB
f memory_get_usage: 1.31 MB
f memory_get_usage: 1.31 MB
f memory_get_usage: 1.31 MB
f memory_get_usage: 1.31 MB
f memory_get_usage: 1.31 MB
f memory_get_usage: 1.31 MB
f memory_get_usage: 1.31 MB
f memory_get_usage: 1.31 MB

BTW I'm using this as command line

php -v

PHP 5.2.17 (cli) (built: Jul 27 2012 08:30:24)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies
with the ionCube PHP Loader v4.0.12, Copyright (c) 2002-2011, by ionCube Ltd., and
with Zend Optimizer v3.3.9, Copyright (c) 1998-2009, by Zend Technologies

Hi again.

any luck to find the root cause, could you guide me to where to find the issue and I will try to fix it and submit a fix

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment