Skip to content

Generate configuration files for apache and nginx #141

Closed
wants to merge 80 commits into from

2 participants

@oschaaf
Google PageSpeed member
oschaaf commented Jan 22, 2013

This pull implements generating apache and nginx configuration files for testing purposes

You can generate an nginx configuration file using:

./cli.py pagespeed.debug.pyconf nginx [directive] > test.nginx.conf

This can be tested using, for example:

/usr/local/nginx/sbin/nginx -t -c <path_to>/test.nginx.conf

An apache configuration file can be generated using

./cli.py pagespeed.debug.pyconf apache [directive] > test.apache.conf
oschaaf added some commits Dec 28, 2012
@oschaaf oschaaf generate-configuration: work in progress
This is a work in progress. The aims to port pagespeed.debug.conf
to a new generic format, from which specific configurations can be
generated for apache, nginx, and others.
a195823
@oschaaf oschaaf generate-configuration: save some work
this saves some work, before refactoring out some common logic
in the templates
1e0125b
@oschaaf oschaaf generate-configuration: save some work 8d84add
@oschaaf oschaaf generate-configuration: save some work
- a more structured approach to rewriting pagespeed.debug.conf
- an alternate approach to translating to an apache configuration
de9fcbf
@oschaaf oschaaf generate-configuration: save some work
- fill in more details
- add new nginx translation
bd4cbc9
@oschaaf oschaaf generate-configuration: handle pagespeed on/off directive 2b2054f
@oschaaf oschaaf generate-configuration: handle all conditionall pagespeed directives a76d40c
@oschaaf oschaaf generate-configuration: rewrite coverage into generic config 1ba7d29
@oschaaf oschaaf generate-configuration: improve output indentation for conditional lines 05a057c
@oschaaf oschaaf generate-configuration: rewrite the remaining virtual hosts 4fcccff
@oschaaf oschaaf generate-configuration: remove global listen directives in nginx output 6eee743
@oschaaf oschaaf generate-configuration: 'inherit' locations/directory directives in n…
…ginx output
2298035
@oschaaf oschaaf generate-configuration: clean up 2158e66
@oschaaf oschaaf generate-configuration: work on pagespeed.debug.pyconf, list todo's 44d6228
@oschaaf oschaaf generate-configuration: gzip,ssl,headers,etc 58b1a23
@oschaaf oschaaf generate-configuration: modules, header actions a7df405
@oschaaf oschaaf generate-configuration: mime-types, apache specific literals, make so…
…me notes
ce9e5d1
@oschaaf oschaaf generate-configuration: merge directory sections w. the same path 15f14d0
@oschaaf oschaaf generate-configuration: merge 2 mod_pagespeed_example/ directory sect…
…ions
ba51472
@oschaaf oschaaf generate-configuration: add more apache-specific literals 3bb5933
@oschaaf oschaaf generate-configuration: handle proxy_pass 25a933a
@oschaaf oschaaf generate-configuration: handle options, rewrite rules 2c90fac
@oschaaf oschaaf generate-configuration: add temporary locations to force pagespeed ha…
…ndler
8e54ff3
@oschaaf oschaaf generate-configuration: style changes 352e1ea
@oschaaf oschaaf generate-configuration: require mime/pagespeed modules 7a5b0fe
@oschaaf oschaaf generate-configuration: make httpd parse the output OK 67f25cb
@oschaaf oschaaf generate-configuration: add lost filecachepaths 93b57c1
@oschaaf oschaaf generate-configuration: add lost filecachepaths 07f7c53
@oschaaf oschaaf generate-configuration: changes after first tests 82e17a9
@oschaaf oschaaf Merge branch 'master' of https://github.com/pagespeed/ngx_pagespeed i…
…nto oschaaf-generate-config
4ce45e9
@oschaaf oschaaf generate-configuration: improve preprocessing
- now allows ifdef/ifndef/endif secions,for easier formatting
- pagespeed options that aren't implemented in nginx yet are
  #ifdefed out in pagespeed.debug.pyconf
- remove two old translations
184caf9
@oschaaf oschaaf generate-configuration: make nginx's configuration test pass 22c0f9f
@oschaaf oschaaf generate-configuration: save state 0f0f9c9
@oschaaf oschaaf load-from-file: save work before switch branch 6113f96
@oschaaf oschaaf generate-confgiguration: handle unarysub in pagespeed.pyconf c0dcd86
@oschaaf oschaaf Merge branch 'master' of https://github.com/pagespeed/ngx_pagespeed i…
…nto oschaaf-generate-config
bdee9e6
@oschaaf oschaaf generate-confgiguration: merge header directives in locations ae9cc6f
@oschaaf oschaaf generate-confgiguration: changes after testing apache/nginx outputs f70c4c6
@oschaaf oschaaf generate-confgiguration: cleanup pagespeed.debug.pyconf 6f40368
@oschaaf oschaaf generate-configuration: apache ssl 4932393
@oschaaf oschaaf generate-configuration: cosmetic changes 081a43b
@oschaaf oschaaf generate-configuration: whitespace 993a578
@oschaaf oschaaf generate-configuration: style fixes ffe5090
@oschaaf oschaaf generate-configuration: more style fixes 9077d19
@oschaaf oschaaf generate-configuration: remove directories directive handling ca089cd
@oschaaf oschaaf generate-configuration: whitespace formatting in templates 0ae7619
@oschaaf oschaaf generate-configuration: strip unused helpers b0b901c
@oschaaf oschaaf generate-configuration: more formatting d69ffa9
@oschaaf oschaaf generate-configuration: process todo's in comments b27e813
@oschaaf oschaaf generate-configuration: save state ff0e5ec
@oschaaf oschaaf generate-configuration: add license banner, consolidate todo's from code 08a6b3a
@oschaaf oschaaf generate-configuration: work through todo list
- add license banners
- move apache_literal to an ifdef'd generic literal
- insert a line in output indicating time and commandline used
- rewrite filesmatch stuff to location matches
666aead
@oschaaf oschaaf generate-configuration: add license banners in templates 41e3c61
@oschaaf oschaaf generate-configuration: improve indent method, style changes 7d83725
@oschaaf oschaaf generate-configuration: update location handling 41ed326
@oschaaf oschaaf generate-configuration: don't hardcode input .pyconf file 7101a7f
@oschaaf oschaaf generate-configuration: update todo's b365a09
@oschaaf oschaaf generate-configuration: process ngx_system_test.sh results 87fda3a
@oschaaf oschaaf Merge remote-tracking branch 'origin/master' into oschaaf-generate-co…
…nfig
3964a73
@oschaaf oschaaf Update README.md 1f67dd5
@jeffkaufman jeffkaufman and 1 other commented on an outdated diff Feb 11, 2013
test/genconf/cli.py
+mode = ""
+
+if len(sys.argv) == 4:
+ input_config_path = sys.argv[1]
+ output_format = sys.argv[2]
+ mode = sys.argv[3]
+else:
+ exit_with_help_message()
+
+conditions[mode] = True
+conditions[output_format] = True
+placeholders["__template_header"] = "generated at %s through \"%s\"" %\
+ (datetime.datetime.now().strftime('%b-%d-%I%M%p-%G'), ' '.join(sys.argv))
+
+#by convention, a file named <outputformat>.conf.template is
+#expected to contain the translation script
@jeffkaufman
Google PageSpeed member
jeffkaufman added a note Feb 11, 2013

spaces before comments

@oschaaf
Google PageSpeed member
oschaaf added a note Feb 12, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jeffkaufman jeffkaufman and 1 other commented on an outdated diff Feb 11, 2013
test/genconf/cli.py
+ mode = sys.argv[3]
+else:
+ exit_with_help_message()
+
+conditions[mode] = True
+conditions[output_format] = True
+placeholders["__template_header"] = "generated at %s through \"%s\"" %\
+ (datetime.datetime.now().strftime('%b-%d-%I%M%p-%G'), ' '.join(sys.argv))
+
+#by convention, a file named <outputformat>.conf.template is
+#expected to contain the translation script
+template = output_format + '.conf.template'
+
+text = execute_template(input_config_path, conditions,
+ placeholders, template)
+print(text)
@jeffkaufman
Google PageSpeed member
jeffkaufman added a note Feb 11, 2013

if this is for python2, which I think it is, no parens needed

@oschaaf
Google PageSpeed member
oschaaf added a note Feb 12, 2013

@jeffkaufman
removed in cb26cb1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jeffkaufman jeffkaufman and 1 other commented on an outdated diff Feb 11, 2013
test/genconf/cli.py
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Author: oschaaf@gmail.com (Otto van der Schaaf)
+
+
+# configuration generator for pagespeed
+
+import datetime
+from genconf import execute_template
+import sys
+from util import Error
+
+def exit_with_help_message():
+ print "This script transforms .pyconf files into webserver configuration files"
+ print "usage: ./genconf.py <input.pyconf> <output_format> <mode>"
@jeffkaufman
Google PageSpeed member
jeffkaufman added a note Feb 11, 2013

./cli.py

@oschaaf
Google PageSpeed member
oschaaf added a note Feb 12, 2013

@jeffkaufman
corrected in 6de3391

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jeffkaufman jeffkaufman and 1 other commented on an outdated diff Feb 11, 2013
test/genconf/cli.py
+# limitations under the License.
+#
+# Author: oschaaf@gmail.com (Otto van der Schaaf)
+
+
+# configuration generator for pagespeed
+
+import datetime
+from genconf import execute_template
+import sys
+from util import Error
+
+def exit_with_help_message():
+ print "This script transforms .pyconf files into webserver configuration files"
+ print "usage: ./genconf.py <input.pyconf> <output_format> <mode>"
+ print "where output_format can be either 'apache' or 'nginx'"
@jeffkaufman
Google PageSpeed member
jeffkaufman added a note Feb 11, 2013

doc what <mode> can be

@oschaaf
Google PageSpeed member
oschaaf added a note Feb 12, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jeffkaufman jeffkaufman commented on an outdated diff Feb 11, 2013
test/genconf/pagespeed.debug.pyconf
+# For the vary: handling test
+ "headers",
+# For the mod_rewrite test in apache_system_test.sh
+ "rewrite",
+#ifdef apache
+#SPELING # Enable mod_speling to ensure that we don't regress Issue 194
+#SPELING "speling",
+#endif
+#HTTPS "ssl",
+#HTTPS "socache_shmcb",
+#HTTPS "slotmem_shm",
+#ifdef nginx
+"gzip",
+#endif
+#ifdef apache
+#GZIP "gzip",
@jeffkaufman
Google PageSpeed member
jeffkaufman added a note Feb 11, 2013

What do you think of supporting syntax like:

#GZIP,apache "gzip",

Having fewer ifdef/endif lines would be more readable.

@jeffkaufman
Google PageSpeed member
jeffkaufman added a note Feb 12, 2013

I made a pull request to change this: #168

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jeffkaufman jeffkaufman commented on an outdated diff Feb 11, 2013
test/genconf/pagespeed.debug.pyconf
+ # However we should not allow user-specified cache-control on
+ # rewritten HTML or resources. This setting helps us make
+ # sure that we strip any user-specified cache-control when
+ # we rewrite HTML. We test this in apache_system_test.sh.
+ {
+ action: "set",
+ name: "Cache-Control",
+ value: "max-age=600",
+ },
+],
+pagespeed: {
+ # If X-PSA-Blocking-Rewrite request header is present and its value matches
+ # the value of ModPagespeedBlockingRewriteKey below, the response will be
+ # fully rewritten before being flushed to the client.
+#ifndef ALL_DIRECTIVES
+ BlockingRewriteKey: "psatest",
@jeffkaufman
Google PageSpeed member
jeffkaufman added a note Feb 11, 2013

What do you think of extending the matching to rephrase these as:

#!ALL_DIRECTIVES BlockingRewriteKey: "psatest",
@jeffkaufman
Google PageSpeed member
jeffkaufman added a note Feb 12, 2013

#168 changes these

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jeffkaufman jeffkaufman and 1 other commented on an outdated diff Feb 11, 2013
test/genconf/pagespeed.debug.pyconf
+ # For regression test of connection failing.
+#ifndef ALL_DIRECTIVES
+ Domain: "modpagespeed.com:1023",
+#endif
+ # Test LoadFromFile mapping by mapping one dir to another.
+
+ LoadFromFile: '"http://@@APACHE_DOMAIN@@/mod_pagespeed_test/load_from_file/web_dir/" "@@APACHE_DOC_ROOT@@/mod_pagespeed_test/load_from_file/file_dir/"',
+ LoadFromFileMatch: '"^http://@@APACHE_DOMAIN@@/mod_pagespeed_test/load_from_file_match/web_([^/]*)/" "@@APACHE_DOC_ROOT@@/mod_pagespeed_test/load_from_file/file_\\1/"',
+ LoadFromFileRule: "Disallow \"@@APACHE_DOC_ROOT@@/mod_pagespeed_test/load_from_file/file_dir/httponly/\"",
+ LoadFromFileRuleMatch: ["Disallow \.ssp.css$","Allow exception\.ssp\.css$"],
+
+ # Print out detail about connection-refused errors. We don't negative-test
+ # this here cause it's a hassle; we just depend on the unit-tests for that.
+ ## not supported on ningx yet (psa r2338):
+ ##ListOutstandingUrlsOnError: "on",
+# Avoid duplicate entry when ALL_DIRECTIVES is specified
@jeffkaufman
Google PageSpeed member
jeffkaufman added a note Feb 11, 2013

Is this actually a problem? Do we need the #ifndef ALL_DIRECTIVES everywhere? They make it really hard to read. What happens if we just include the directive twice? Or move ALL_DIRECTIVES into its own vhost or something? Or even wrap everything in an giant #ifndef ALL_DIRECTIVES except and then a separate #ifdef ALL_DIRECTIVES at the end?

@oschaaf
Google PageSpeed member
oschaaf added a note Feb 12, 2013

@jeffkaufman
Currently, including the a directive with the same name twice will result in an error noting that it is already defined ( an array of values can be used to specify multiple values for the same directive instead).

Moving the ALL_DIRECTIVES to a separate vhost sounds like a good idea to me, do you want me to do that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jeffkaufman jeffkaufman and 1 other commented on an outdated diff Feb 11, 2013
test/genconf/pagespeed.debug.pyconf
+ #STATS_LOGGING StatisticsLogging: "on",
+ #STATS_LOGGING StatisticsLoggingFile: "\"@@MOD_PAGESPEED_STATS_LOG@@\"",
+ #STATS_LOGGING StatisticsLoggingIntervalMs: 10,
+#ifdef ALL_DIRECTIVES
+ # Invoke all ModPagespeed* directives to make sure they work:
+ Allow: "foo",
+ AnalyticsID: 1234,
+ AvoidRenamingIntrospectiveJavascript: "true",
+ BeaconUrl: "\"http://example.com/beacon\"",
+ BlockingRewriteKey: "test",
+#ifndef nginx
+ CacheFlushFilename: "/tmp/cache.flush",
+ CacheFlushPollInterevalSec: 10,
+#endif
+ ClientDomainRewrite: "false",
+#ifndef nginx
@jeffkaufman
Google PageSpeed member
jeffkaufman added a note Feb 11, 2013

I think it would be better to filter these out later. The less the shared code has to know what nginx does and doesn't support the better. What if we made the pyconf-to-nginx code have a list of unsupported directives that it would spit out a warning for (stripping them out)?

@oschaaf
Google PageSpeed member
oschaaf added a note Feb 12, 2013

@jeffkaufman

I'll look in to it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jeffkaufman
Google PageSpeed member

To test the config system I ran:

for server in apache nginx ; do for mode in ALL_DIRECTIVES COVERAGE FURIOUS_GA FURIOUS_NO_GA GZIP HTTPS MEMCACHE_COVERAGE MEMCACHED PER_VHOST_STATS PROXY REWRITE SHARED_MEM_LOCKS SLURP SPELING STATS_LOGGING STRESS ; do CMD="python cli.py pagespeed.debug.pyconf $server $mode" ; $CMD > $server.$mode.conf || echo $CMD ; done ; done

This turned up some simple errors, which I fixed: #169

Two errors remain which weren't easily fixed:

$ python cli.py pagespeed.debug.pyconf nginx PER_VHOST_STATS
Traceback (most recent call last):
  File "cli.py", line 81, in <module>
    placeholders, template)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/genconf.py", line 244, in execute_template
    text = template.render(config=config)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/templite.py", line 81, in render
    eval(self.__code, namespace)
  File "<templite '${\n# Copyright 2013 '>", line 222, in <module>
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 38, in write_cfg
    write_cfg(key_to_writer, item, level + 1)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 38, in write_cfg
    write_cfg(key_to_writer, item, level + 1)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 51, in write_cfg
    raise Error("no tranform handler for [%s]" % key)
util.Error: no tranform handler for [literal]

$ python cli.py pagespeed.debug.pyconf nginx STRESS
Traceback (most recent call last):
  File "cli.py", line 81, in <module>
    placeholders, template)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/genconf.py", line 244, in execute_template
    text = template.render(config=config)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/templite.py", line 81, in render
    eval(self.__code, namespace)
  File "<templite '${\n# Copyright 2013 '>", line 222, in <module>
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 38, in write_cfg
    write_cfg(key_to_writer, item, level + 1)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 38, in write_cfg
    write_cfg(key_to_writer, item, level + 1)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 30, in write_cfg
    handled = w(config[key], level)
  File "<templite '${\n# Copyright 2013 '>", line 168, in write_add_options_open
Exception: Option 'ExecCGI': no translation for nginx

It would be nice if errors could be accompanied by the line number the error's on.

@jeffkaufman
Google PageSpeed member

Moving the ALL_DIRECTIVES to a separate vhost sounds like a good idea to me, do you want me to do that?

That would be great. Or, if you'd rather, I can do it.

jeffkaufman and others added some commits Feb 12, 2013
@jeffkaufman jeffkaufman Merge pull request #169 from pagespeed/jefftk-generate-config-verific…
…ation

fix some template errors
54f3560
@oschaaf oschaaf generate-config: move ALL_DIRECTIVES to a separate vhost 959272e
@oschaaf oschaaf generate-config: restore apache conditions 1d5755b
@jeffkaufman jeffkaufman Merge pull request #171 from pagespeed/oschaaf-generate-config-all-di…
…rectives

Move ALL_DIRECTIVES into its own VHOST
ea4582f
@oschaaf oschaaf generate-config: display line numbers in errors 455a9bb
@oschaaf
Google PageSpeed member
oschaaf commented Feb 13, 2013

@jeffkaufman
"It would be nice if errors could be accompanied by the line number the error's on."

That should be very useful, but it's a little harder then I thought initially, because we have semi-inheritance for locations and header directives in .pyconf.
To be able to make a best effort to display line numbers, the script needs to somehow track the source of these properties as they get inherited. I am looking in to that.

oschaaf and others added some commits Feb 13, 2013
@oschaaf oschaaf generate-config: line numbers in error messages b680348
@oschaaf oschaaf generate-config: tidy up 58ec586
@jeffkaufman jeffkaufman Merge pull request #172 from pagespeed/oschaaf-generate-config-linenu…
…mbers

generate-config: display line numbers in errors
a22f1f8
@jeffkaufman jeffkaufman commented on the diff Feb 13, 2013
test/genconf/pagespeed.debug.pyconf
+#ifdef apache
+ literal: {
+ value: """
+ Order allow,deny
+ Allow from localhost
+ Allow from 127.0.0.1
+ SetHandler mod_pagespeed_global_statistics\n""",
+ }
+#endif
+ },
+#endif
+#ifdef STRESS
+ # These lines are only needed for the stress test.
+ {
+ path: "/mod_pagespeed_example/cgi",
+ add_options: ["ExecCGI"],
@jeffkaufman
Google PageSpeed member
jeffkaufman added a note Feb 13, 2013

This is apache-only

@oschaaf
Google PageSpeed member
oschaaf added a note Feb 14, 2013

@jeffkaufman: "This is apache-only"
Do you think it would be feasible to have something like http://wiki.nginx.org/SimpleCGI generated for nginx at this point?

@jeffkaufman
Google PageSpeed member
jeffkaufman added a note Feb 14, 2013

Probably. For now I'd be inclined to mark it as apache only, and then as I start going through apache_system_test.sh I can fix more and more of the apache-only config stuff to be general.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jeffkaufman
Google PageSpeed member

mixes tabs and spaces

Google PageSpeed member

fixed in 8083e10

oschaaf and others added some commits Feb 14, 2013
@oschaaf oschaaf generate-config: don't mix tabs and spaces 8083e10
@jeffkaufman jeffkaufman Merge pull request #174 from pagespeed/oschaaf-generate-config-unsupp…
…orted-directives

generate config: don't filter unsupported pagespeed directives in .pyconf files
46e1e48
@jeffkaufman
Google PageSpeed member

Next steps:

1) I'm going to bring this inside and modify mod_pagespeed's net/instaweb/apache/install/Makefile* to use genconf.

2) Once that's working, I'll refactor the makefiles to get a big generic chunk and a small apache-specific chunk

3) Then, back on github, we can write the nginx-specific chunk

And then, with the system tests running, we'll have lots of bugs to look into.

@jeffkaufman
Google PageSpeed member

When this comes in it's coming via the mod_pagespeed codebase, not via here.

@jeffkaufman jeffkaufman deleted the oschaaf-generate-config branch Mar 11, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.