LoadFromFile blacklist support

[GoogleCodeExporter]

Consider a site that is static html with a dynamic directory:

  /index.html
  /about.html
  /images/
  /cgi-bin/
  /stylesheets/
  /scripts/
  /subdirA/
  /subdirB/
  /subdirC/
  ...

They would like to set:

  ModPagespeedLoadFromFile "http://www.example.com/" "/var/www/"

But this would break their cgi-bin directory.  While they could add separate 
"ModPagespeedLoadFromFile" directives for every top-level directory except 
cgi-bin, that is awkward, needs to be maintained, doesn't apply to index.html 
or other top-level pages, and may slow mod_pagespeed down.  Finer-grained 
control over LoadFromFile is needed here.

I see two potential solutions, of which I prefer (1):

  1) Add a "ModPagespeedDisableLoadFromFile"
     configuration directive that takes
     priority over ModPagespeedLoadFromFile:

       ModPagespeedLoadFromFile "http://www.example.com/" "/var/www/"
       ModPagespeedDisableLoadFromFile "http://www.example.com/cgi-bin/"

  2) Support directory-level configuration,
     with both a ModPagespeedLoadFromFile and
     a ModPagespeedDisableLoadFromFile.

       <Directory /var/www>
          ModPagespeedLoadFromFile
       </Directory>

       <Directory /var/www/cgi-bin>
          ModPagespeedDisableLoadFromFile
       </Directory>

     In the example of someone with a mostly
     dynamic website that has a /static
     directory you would have:

       <Directory /var/www/static>
          ModPagespeedLoadFromFile
       </Directory>

     (Not loading from file is the default.)

One question is how useful this is: how many sites does it ally to?  While 
traditionally websites were set up this way, dynamic sites may be the default 
now.  Many dynamic sites, however, are actually serving static content that has 
been "baked" in advance.  For example, Movable Type does this by default.

Original issue reported on code.google.com by jefftk@google.com on 11 Jul 2012 at 6:37

[GoogleCodeExporter]

Hm, version #2 has the nice property that it takes advantage of standard Apache 
scopes and can thus be applied anywhere other Apache commands can be. I wonder 
if we could do something similar for issues like issue 377 where people want to 
use some (perhaps Apache config defined) variable in their directory name.

Original comment by sligocki@google.com on 12 Jul 2012 at 6:27

[GoogleCodeExporter]

The problem with using the <Directory> scope to accomplish this is that we 
don't get the directory-specific RewriteOptions for the resource when rewriting 
HTML.  We get the directory-specific RewriteOptions for the HTML file.

Original comment by jmara...@google.com on 12 Jul 2012 at 6:29

[GoogleCodeExporter]

Oh, yes, that is bad.

Original comment by sligocki@google.com on 12 Jul 2012 at 6:33

[GoogleCodeExporter]

Fixed in revision 1835.  Documentation to come, but the syntax is:

   ModPagespeedLoadFromFile http://example.com/ /var/www/
   ModPagespeedLoadFromFileRule Disallow /var/www/cgi-bin/
   ModPagespeedLoadFromFileRuleMatch Allow \.css$

The first argument is either "Allow" or "Disallow", the second is a literal 
prefix for ModPagespeedLoadFromFileRule and a RE2-syntax regular expression for 
ModPagespeedLoadFromFileRuleMatch.  By default anything mapped with 
LoadFromFile is allowed to be loaded from file (in this case: all of /var/www/) 
and then you add rules to adjust loading permissions from there.  The first 
rule, a Disallow, changes our configuration to "load everything from file for 
http://example.com/ except things we would find in /var/www/cgi-bin/ (so 
http://example.com/cgi-bin/...)" and the second rule, the Allow, partially 
undoes that to permit loading of files with extension ".css" even if they're 
under /var/www/cgi-bin/.

Original comment by jefftk@google.com on 30 Aug 2012 at 2:40

• Changed state: Fixed

[GoogleCodeExporter]

Original comment by sligocki@google.com on 30 Aug 2012 at 2:47

• Added labels: release-note

[GoogleCodeExporter]

Original comment by j...@google.com on 26 Oct 2012 at 5:57

• Added labels: Milestone-v23