Extreme memory usage.

[zetas]

For some reason rarinfo is using insane amounts of memory. It actually used up all free memory on my main server that has 256gb of ram. I reduced the ram usage allotment for php to 4gb to keep it from crashing the entire server but now all I see in my error log is this:

[05-Jul-2016 11:56:36 America/New_York] PHP Fatal error: Allowed memory size of 4294967296 bytes exhausted (tried to allocate 72 bytes) in /libs/zeebinz/rarinfo/rarinfo.php on line 1287 [05-Jul-2016 11:56:37 America/New_York] PHP Fatal error: Allowed memory size of 4294967296 bytes exhausted (tried to allocate 32 bytes) in zeebinz/rarinfo/rarinfo.php on line 1404

Over and over and over again, different line numbers all the time but hovering between line 1200 and 1450 or so. When this fatal error occurs it actually crashes the php-fpm thread it was running on and any other processes that were using it fail and the end-user gets a white screen for that page.

[zeebinz]

Do you have a sample of the RAR file causing the problem?

[DariusIII]

Main cause is $block['extra_size'] having an extremely high value https://github.com/zeebinz/rarinfo/blob/master/rarinfo.php#L1277. So the while loop is iterating until it is exhausting all of the available memory.

I have var_dumped the hell out of it, and $block['extra_size'] on proper RAR goes from 9 to 12 (on samples used) while on bad RARs value is sometimes even 2-3M so the $end value ($this->offset+$block['extra_size']) is very large value while loop needs to iterate on.

[LeeThompson]

I'm seeing this too, in newznab.

This is with PHP 7.0.1 x64 with memory limit set to 3GB.

[28-Sep-2016 17:05:13 America/Los_Angeles] PHP Fatal error:  Allowed memory size of 3221225472 bytes exhausted (tried to allocate 20480 bytes) in <redacted>\newznab\www\lib\rarinfo\rarinfo.php on line 1282
[28-Sep-2016 17:08:50 America/Los_Angeles] PHP Fatal error:  Allowed memory size of 3221225472 bytes exhausted (tried to allocate 4096 bytes) in <redacted>\newznab\www\lib\rarinfo\rarinfo.php on line 1282

Given that this hasn't been updated in years doesn't give me a lot of hope for a fix.

I'll try to see if I can't find a sample RAR that casues the issue.

[thezoggy]

you trying to index boneless or something ? aka you sure this is fault of rarinfo or just rarinfo running into the fact that theres no free memory... as OOM issues have plagued newznab forever due to badly configured settings (keeping parts) / trying to index really crappy groups and with obfuscation / etc.. which have nothing to do with rarinfo directly... usual fix is to find the 'release' thats causing all the usage and just to nuke it and move along..

the two latest culprits for example are:
delete from releases where gid = '6927a4d3f9a2ba3b9a40c48d5a8d092c';
and
delete from releases where guid = '3e1634ffe6ad3a5f7651a10fd7e37103';

[LeeThompson]

I have a reproducible case. newznab (and probably other indexers as well) often download just a segment of a post and then try to identify the file contents (rarinfo and other tools are used). This particular file, for some reason, causes rarinfo to allocate memory until php terminates it.

File sample: sample file link

Test code:
(Adjust paths as needed.)

<?php
require_once("rarinfo/archiveinfo.php");
$filename = "bin/7244026_20160929184238.bin";
echo "Testing with $filename\n";
$fetchedbinary = file_get_contents($filename);
echo "Loaded Size: " . strlen($fetchedbinary) . "\n";
$rar = new ArchiveInfo();
$rar->setData($fetchedbinary, true);
if ($rar->error) {
    echo "Error reading file.\n";
} else {
    echo "Attempting getArchiveFileList\n";
    $files = $rar->getArchiveFileList();
    if ($files !== false) {
        echo "Files:\n";
        foreach ($files as $file) {
            if (isset($file['name'])) {
                echo "File: " . $file['name'] . "\n";
            }
        }
    } else {
        echo "Failed to get file list.\n";
    }
}
?>

[thezoggy]

what happens if you call $files = $rar->getArchiveFileList(); with recursive being false?
also your shared file doesnt work

[LeeThompson]

If I do $files = $rar->getArchiveFileList(false); I get the same behavior.

if ($files !== false) is what's in newznab's code. Not that it gets that far with this case.

Sample should work now.

[LeeThompson]

I saw earlier in the thread concern about $block['extra_size'] possibly becoming too large, that does not seem to be the case.

I added an echo for some additional data to be shown to rarinfo.php:
echo "processExtraRecords (offset: $this->offset < end: $end), block(extra_size): " . $block['extra_size'] . "\n";

Result:

Testing with bin/7244026_20160929184238.bin
Loaded Size: 384000
processExtraRecords (offset: 17 < end: 25), block(extra_size): 8
processExtraRecords (offset: 1782 < end: 1793), block(extra_size): 11
processExtraRecords (offset: 364211 < end: 364222), block(extra_size): 11
processExtraRecords (offset: 384000 < end: 384011), block(extra_size): 11

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 24 bytes) in C:\Source\_debug\rarinfo\rarinfo.php on line 1401

[thezoggy]

are you using nzedb/darrius's version?
https://github.com/DariusIII/rarinfo

which they had to update to support php 7.x...

[LeeThompson]

I'm using whatever is packaged with newznab 0.2.3 r3505.

I have been running PHP 5.5.30 (when this issue first arose as well), I just recently upgraded to PHP 7. I haven't had any other problems than this.

I'll pass along the rarinfo fork info to the newznab people though.

[thezoggy]

mind trying out darius's fork to see if it has the same issue?

[DariusIII]

My fork is having same issues, but feel free to try it.

[LeeThompson]

I'm having trouble testing DarisIII's fork since it seems to use a different namespace.

Basically I get

Fatal error: Class 'dariusiii\rarinfo\ArchiveReader' not found in C:\Source\_debug\rarinfo\darius\ArchiveInfo.php on line 72

My ugly test code has it organized like this:

require_once("rarinfo/darius/archiveinfo.php");
# require_once("rarinfo/newznab/archiveinfo.php");
# require_once("rarinfo/zeebinz/archiveinfo.php");

The other two 'forks' (I am pretty sure what newznabhas is identical to the current zeebinz
release).

[DariusIII]

You can't use my fork without composer or PSR-0 autoloader.

[LeeThompson]

I've been doing some debugging with this on the main (zeebinz) branch and the problem may be this.

When the memory allocation error occurs, in rarinfo.php the function processExtraRecords is called as usual, however, in this particular case, the initial offset is already at the end of the fragment passed in.

In this example, when setDatais called at the start of the process, the fragment is 384000 bytes.

rarinfo/processExtraRecords is called, the offset is already at 384000, it adds $block['extra_size'] (which contains 11) and $end is set to 384011.

The while loop is entered with the condition while (384000 < 384011) and as soon as it tries do load $rec array's size value with $size = $this->getVarInt(), and eventually memory_limit is hit.

It goes into getVarInt and immediately drops out of the while loop since the condition$this->offset < $this->length && $count <= 10 is already met and so does return 0;

rarinfo/getVarInt
rarinfo/getVarInt/while (384000 < 384000 && 1 <= 10) start
rarinfo/getVarInt/while end(0)

For some reason (PHP bug?), the part in marked below keeps executing over and over:

$rec = array(
    'name'   => 'Unknown',
    'offset' => $this->offset,
    'size'   => $size = $this->getVarInt(),    // Loops On This
    'next'   => $this->offset + $size,
    'type'   => $this->getVarInt(),
);

The solution perhaps would be to check $this->offset against $this->length to make sure it's not about to read out of range data.

[LeeThompson]

I've made a couple patches to check$this->offset against $this->length when setting $end. I've tested it on a half dozen archives (zip and rar) and so far it works but should probably be tested more thoroughly. :)

fix_issue_9.zip

[zeebinz]

Thanks @LeeThompson for the sample file and digging into the code. Yes, the problem was with certain corrupt headers and variable ints, happily the fix was an easy one.