Repository navigation
LoadFromFile crashes on resources too big for memory #1386
Description
Activity
Definitely we should be statting files and comparing size against some max value before allocating anything. I think we should be comparing against the max cacheable size, and not an image-specific size.
Since we do cache extend PDFs I think it does make sense for them to work with LoadFromFile. But I don't think it makes sense to do that in the IPRO flow, only the extend_cache filter. We might need to plumb through some context to FileInputResource to make that judgement call, and just skip file-loading .pdfs from IPRO.
We have a
MaxCacheableContentLength, but that defaults to unlimited. Looking briefly, I don't see any other general limits on how big a resource we can optimize.I'm thinking we should switch that default from unlimited to something high but not crazy, like 32MB, and then use it in LFF?
@hillsp What do you think? Should we lower the existing limit, or add a new limit?
Do we do any streaming re-writing? If not, I'm inclined to go with using the existing option.
In theory we could do streaming minification of text formats, but nothing
at all is actually designed for that to work, and text is rarely large
enough for that to be worth it.I'll plan to use the existing option then.
- added a commit that references this issue
on Sep 9, 2016 Repro link is no longer a query-of-death for jefftk.com, since I manually blacklisted LoadFromFile for pdf files:
pagespeed LoadFromFileRuleMatch Disallow \.pdf$;Is this not still a problem for resources loaded via http as well? That is, doesn't your server now just loopback fetch the pdf file and have the same memory problems?
doesn't your server now just loopback fetch the pdf file and have the same memory problems?
We don't loopback fetch for IPRO, and we do abort recording if the content-length header is too large.
but wouldn't extend_cache_pdfs trigger a loopback fetch? Or do you have have that disabled on your server?
Also is
pagespeed Disallow *.pdf
a workaround for this?extend_cache_pdfs is off by default, and I haven't enabled it.
Disallowing pdfs globally would work, and is slightly better than what I did which was just disabling pdfs for LoadFromFile.
- changed the title
[-]Failure on very large PDF with LoadFromFile[/-][+]LoadFromFile crashes on resources too big for memory[/+]on Oct 11, 2016 - added a commit that references this issue
on Nov 9, 2025
Here's a large (251MB) PDF: http://www.jefftk.com/mike-mulligan-obsolete-internal.pdf?PageSpeed=off
With PageSpeed on, it doesn't load: http://www.jefftk.com/mike-mulligan-obsolete-internal.pdf
This is with LoadFromFile on an ngx_pagespeed system without much memory.
Possibly we're blindly trying to allocate enough memory for it, without checking its size? Yes, I think that's it: neither
FileInputResource::LoadAndCallback()norFileSystem::ReadFile()check the file size and refuse to load things that are too big. Anything overImageResolutionLimitBytes(which is for uncompressed images, and real images will be much smaller) makes no sense to load.It's also surprising that we'd be doing this for a PDF, since we can't optimize that and we know that immediately from the extension. Looking at
FileLoadPolicy::ShouldLoadFromFile, though, our rule for which content types to load from file isn't "can we optimize it" but "do we know what this extension means" + "is it a static resource type".