Generate configuration files for apache and nginx #141

Closed
wants to merge 80 commits into
from

Conversation

Projects
None yet
2 participants
@oschaaf
Member

oschaaf commented Jan 22, 2013

This pull implements generating apache and nginx configuration files for testing purposes

You can generate an nginx configuration file using:

./cli.py pagespeed.debug.pyconf nginx [directive] > test.nginx.conf

This can be tested using, for example:

/usr/local/nginx/sbin/nginx -t -c <path_to>/test.nginx.conf

An apache configuration file can be generated using

./cli.py pagespeed.debug.pyconf apache [directive] > test.apache.conf

oschaaf added some commits Dec 28, 2012

generate-configuration: work in progress
This is a work in progress. The aims to port pagespeed.debug.conf
to a new generic format, from which specific configurations can be
generated for apache, nginx, and others.
generate-configuration: save some work
this saves some work, before refactoring out some common logic
in the templates
generate-configuration: save some work
- a more structured approach to rewriting pagespeed.debug.conf
- an alternate approach to translating to an apache configuration
generate-configuration: save some work
- fill in more details
- add new nginx translation
generate-configuration: improve preprocessing
- now allows ifdef/ifndef/endif secions,for easier formatting
- pagespeed options that aren't implemented in nginx yet are
  #ifdefed out in pagespeed.debug.pyconf
- remove two old translations
test/genconf/cli.py
+ (datetime.datetime.now().strftime('%b-%d-%I%M%p-%G'), ' '.join(sys.argv))
+
+#by convention, a file named <outputformat>.conf.template is
+#expected to contain the translation script

This comment has been minimized.

@jeffkaufman

jeffkaufman Feb 11, 2013

Contributor

spaces before comments

@jeffkaufman

jeffkaufman Feb 11, 2013

Contributor

spaces before comments

This comment has been minimized.

@oschaaf

oschaaf Feb 12, 2013

Member
test/genconf/cli.py
+
+text = execute_template(input_config_path, conditions,
+ placeholders, template)
+print(text)

This comment has been minimized.

@jeffkaufman

jeffkaufman Feb 11, 2013

Contributor

if this is for python2, which I think it is, no parens needed

@jeffkaufman

jeffkaufman Feb 11, 2013

Contributor

if this is for python2, which I think it is, no parens needed

This comment has been minimized.

@oschaaf

oschaaf Feb 12, 2013

Member

@jeffkaufman
removed in cb26cb1

test/genconf/cli.py
+
+def exit_with_help_message():
+ print "This script transforms .pyconf files into webserver configuration files"
+ print "usage: ./genconf.py <input.pyconf> <output_format> <mode>"

This comment has been minimized.

@jeffkaufman

jeffkaufman Feb 11, 2013

Contributor

./cli.py

@jeffkaufman

jeffkaufman Feb 11, 2013

Contributor

./cli.py

This comment has been minimized.

@oschaaf

oschaaf Feb 12, 2013

Member

@jeffkaufman
corrected in 6de3391

test/genconf/cli.py
+def exit_with_help_message():
+ print "This script transforms .pyconf files into webserver configuration files"
+ print "usage: ./genconf.py <input.pyconf> <output_format> <mode>"
+ print "where output_format can be either 'apache' or 'nginx'"

This comment has been minimized.

@jeffkaufman

jeffkaufman Feb 11, 2013

Contributor

doc what <mode> can be

@jeffkaufman

jeffkaufman Feb 11, 2013

Contributor

doc what <mode> can be

This comment has been minimized.

@oschaaf

oschaaf Feb 12, 2013

Member
test/genconf/pagespeed.debug.pyconf
+"gzip",
+#endif
+#ifdef apache
+#GZIP "gzip",

This comment has been minimized.

@jeffkaufman

jeffkaufman Feb 11, 2013

Contributor

What do you think of supporting syntax like:

#GZIP,apache "gzip",

Having fewer ifdef/endif lines would be more readable.

@jeffkaufman

jeffkaufman Feb 11, 2013

Contributor

What do you think of supporting syntax like:

#GZIP,apache "gzip",

Having fewer ifdef/endif lines would be more readable.

This comment has been minimized.

@jeffkaufman

jeffkaufman Feb 12, 2013

Contributor

I made a pull request to change this: pagespeed#168

@jeffkaufman

jeffkaufman Feb 12, 2013

Contributor

I made a pull request to change this: pagespeed#168

test/genconf/pagespeed.debug.pyconf
+ # the value of ModPagespeedBlockingRewriteKey below, the response will be
+ # fully rewritten before being flushed to the client.
+#ifndef ALL_DIRECTIVES
+ BlockingRewriteKey: "psatest",

This comment has been minimized.

@jeffkaufman

jeffkaufman Feb 11, 2013

Contributor

What do you think of extending the matching to rephrase these as:

#!ALL_DIRECTIVES BlockingRewriteKey: "psatest",
@jeffkaufman

jeffkaufman Feb 11, 2013

Contributor

What do you think of extending the matching to rephrase these as:

#!ALL_DIRECTIVES BlockingRewriteKey: "psatest",

This comment has been minimized.

@jeffkaufman

jeffkaufman Feb 12, 2013

Contributor

pagespeed#168 changes these

@jeffkaufman

jeffkaufman Feb 12, 2013

Contributor

pagespeed#168 changes these

test/genconf/pagespeed.debug.pyconf
+ # this here cause it's a hassle; we just depend on the unit-tests for that.
+ ## not supported on ningx yet (psa r2338):
+ ##ListOutstandingUrlsOnError: "on",
+# Avoid duplicate entry when ALL_DIRECTIVES is specified

This comment has been minimized.

@jeffkaufman

jeffkaufman Feb 11, 2013

Contributor

Is this actually a problem? Do we need the #ifndef ALL_DIRECTIVES everywhere? They make it really hard to read. What happens if we just include the directive twice? Or move ALL_DIRECTIVES into its own vhost or something? Or even wrap everything in an giant #ifndef ALL_DIRECTIVES except and then a separate #ifdef ALL_DIRECTIVES at the end?

@jeffkaufman

jeffkaufman Feb 11, 2013

Contributor

Is this actually a problem? Do we need the #ifndef ALL_DIRECTIVES everywhere? They make it really hard to read. What happens if we just include the directive twice? Or move ALL_DIRECTIVES into its own vhost or something? Or even wrap everything in an giant #ifndef ALL_DIRECTIVES except and then a separate #ifdef ALL_DIRECTIVES at the end?

This comment has been minimized.

@oschaaf

oschaaf Feb 12, 2013

Member

@jeffkaufman
Currently, including the a directive with the same name twice will result in an error noting that it is already defined ( an array of values can be used to specify multiple values for the same directive instead).

Moving the ALL_DIRECTIVES to a separate vhost sounds like a good idea to me, do you want me to do that?

@oschaaf

oschaaf Feb 12, 2013

Member

@jeffkaufman
Currently, including the a directive with the same name twice will result in an error noting that it is already defined ( an array of values can be used to specify multiple values for the same directive instead).

Moving the ALL_DIRECTIVES to a separate vhost sounds like a good idea to me, do you want me to do that?

test/genconf/pagespeed.debug.pyconf
+ CacheFlushPollInterevalSec: 10,
+#endif
+ ClientDomainRewrite: "false",
+#ifndef nginx

This comment has been minimized.

@jeffkaufman

jeffkaufman Feb 11, 2013

Contributor

I think it would be better to filter these out later. The less the shared code has to know what nginx does and doesn't support the better. What if we made the pyconf-to-nginx code have a list of unsupported directives that it would spit out a warning for (stripping them out)?

@jeffkaufman

jeffkaufman Feb 11, 2013

Contributor

I think it would be better to filter these out later. The less the shared code has to know what nginx does and doesn't support the better. What if we made the pyconf-to-nginx code have a list of unsupported directives that it would spit out a warning for (stripping them out)?

This comment has been minimized.

@oschaaf

oschaaf Feb 12, 2013

Member

@jeffkaufman

I'll look in to it

@oschaaf

oschaaf Feb 12, 2013

Member

@jeffkaufman

I'll look in to it

@jeffkaufman

This comment has been minimized.

Show comment
Hide comment
@jeffkaufman

jeffkaufman Feb 12, 2013

Contributor

To test the config system I ran:

for server in apache nginx ; do for mode in ALL_DIRECTIVES COVERAGE FURIOUS_GA FURIOUS_NO_GA GZIP HTTPS MEMCACHE_COVERAGE MEMCACHED PER_VHOST_STATS PROXY REWRITE SHARED_MEM_LOCKS SLURP SPELING STATS_LOGGING STRESS ; do CMD="python cli.py pagespeed.debug.pyconf $server $mode" ; $CMD > $server.$mode.conf || echo $CMD ; done ; done

This turned up some simple errors, which I fixed: pagespeed#169

Two errors remain which weren't easily fixed:

$ python cli.py pagespeed.debug.pyconf nginx PER_VHOST_STATS
Traceback (most recent call last):
  File "cli.py", line 81, in <module>
    placeholders, template)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/genconf.py", line 244, in execute_template
    text = template.render(config=config)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/templite.py", line 81, in render
    eval(self.__code, namespace)
  File "<templite '${\n# Copyright 2013 '>", line 222, in <module>
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 38, in write_cfg
    write_cfg(key_to_writer, item, level + 1)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 38, in write_cfg
    write_cfg(key_to_writer, item, level + 1)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 51, in write_cfg
    raise Error("no tranform handler for [%s]" % key)
util.Error: no tranform handler for [literal]

$ python cli.py pagespeed.debug.pyconf nginx STRESS
Traceback (most recent call last):
  File "cli.py", line 81, in <module>
    placeholders, template)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/genconf.py", line 244, in execute_template
    text = template.render(config=config)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/templite.py", line 81, in render
    eval(self.__code, namespace)
  File "<templite '${\n# Copyright 2013 '>", line 222, in <module>
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 38, in write_cfg
    write_cfg(key_to_writer, item, level + 1)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 38, in write_cfg
    write_cfg(key_to_writer, item, level + 1)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 30, in write_cfg
    handled = w(config[key], level)
  File "<templite '${\n# Copyright 2013 '>", line 168, in write_add_options_open
Exception: Option 'ExecCGI': no translation for nginx

It would be nice if errors could be accompanied by the line number the error's on.

Contributor

jeffkaufman commented Feb 12, 2013

To test the config system I ran:

for server in apache nginx ; do for mode in ALL_DIRECTIVES COVERAGE FURIOUS_GA FURIOUS_NO_GA GZIP HTTPS MEMCACHE_COVERAGE MEMCACHED PER_VHOST_STATS PROXY REWRITE SHARED_MEM_LOCKS SLURP SPELING STATS_LOGGING STRESS ; do CMD="python cli.py pagespeed.debug.pyconf $server $mode" ; $CMD > $server.$mode.conf || echo $CMD ; done ; done

This turned up some simple errors, which I fixed: pagespeed#169

Two errors remain which weren't easily fixed:

$ python cli.py pagespeed.debug.pyconf nginx PER_VHOST_STATS
Traceback (most recent call last):
  File "cli.py", line 81, in <module>
    placeholders, template)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/genconf.py", line 244, in execute_template
    text = template.render(config=config)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/templite.py", line 81, in render
    eval(self.__code, namespace)
  File "<templite '${\n# Copyright 2013 '>", line 222, in <module>
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 38, in write_cfg
    write_cfg(key_to_writer, item, level + 1)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 38, in write_cfg
    write_cfg(key_to_writer, item, level + 1)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 51, in write_cfg
    raise Error("no tranform handler for [%s]" % key)
util.Error: no tranform handler for [literal]

$ python cli.py pagespeed.debug.pyconf nginx STRESS
Traceback (most recent call last):
  File "cli.py", line 81, in <module>
    placeholders, template)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/genconf.py", line 244, in execute_template
    text = template.render(config=config)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/templite.py", line 81, in render
    eval(self.__code, namespace)
  File "<templite '${\n# Copyright 2013 '>", line 222, in <module>
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 38, in write_cfg
    write_cfg(key_to_writer, item, level + 1)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 38, in write_cfg
    write_cfg(key_to_writer, item, level + 1)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 30, in write_cfg
    handled = w(config[key], level)
  File "<templite '${\n# Copyright 2013 '>", line 168, in write_add_options_open
Exception: Option 'ExecCGI': no translation for nginx

It would be nice if errors could be accompanied by the line number the error's on.

@jeffkaufman

This comment has been minimized.

Show comment
Hide comment
@jeffkaufman

jeffkaufman Feb 12, 2013

Contributor

Moving the ALL_DIRECTIVES to a separate vhost sounds like a good idea to me, do you want me to do that?

That would be great. Or, if you'd rather, I can do it.

Contributor

jeffkaufman commented Feb 12, 2013

Moving the ALL_DIRECTIVES to a separate vhost sounds like a good idea to me, do you want me to do that?

That would be great. Or, if you'd rather, I can do it.

@oschaaf

This comment has been minimized.

Show comment
Hide comment
@oschaaf

oschaaf Feb 13, 2013

Member

@jeffkaufman
"It would be nice if errors could be accompanied by the line number the error's on."

That should be very useful, but it's a little harder then I thought initially, because we have semi-inheritance for locations and header directives in .pyconf.
To be able to make a best effort to display line numbers, the script needs to somehow track the source of these properties as they get inherited. I am looking in to that.

Member

oschaaf commented Feb 13, 2013

@jeffkaufman
"It would be nice if errors could be accompanied by the line number the error's on."

That should be very useful, but it's a little harder then I thought initially, because we have semi-inheritance for locations and header directives in .pyconf.
To be able to make a best effort to display line numbers, the script needs to somehow track the source of these properties as they get inherited. I am looking in to that.

+ # These lines are only needed for the stress test.
+ {
+ path: "/mod_pagespeed_example/cgi",
+ add_options: ["ExecCGI"],

This comment has been minimized.

@jeffkaufman

jeffkaufman Feb 13, 2013

Contributor

This is apache-only

@jeffkaufman

jeffkaufman Feb 13, 2013

Contributor

This is apache-only

This comment has been minimized.

@oschaaf

oschaaf Feb 14, 2013

Member

@jeffkaufman: "This is apache-only"
Do you think it would be feasible to have something like http://wiki.nginx.org/SimpleCGI generated for nginx at this point?

@oschaaf

oschaaf Feb 14, 2013

Member

@jeffkaufman: "This is apache-only"
Do you think it would be feasible to have something like http://wiki.nginx.org/SimpleCGI generated for nginx at this point?

This comment has been minimized.

@jeffkaufman

jeffkaufman Feb 14, 2013

Contributor

Probably. For now I'd be inclined to mark it as apache only, and then as I start going through apache_system_test.sh I can fix more and more of the apache-only config stuff to be general.

@jeffkaufman

jeffkaufman Feb 14, 2013

Contributor

Probably. For now I'd be inclined to mark it as apache only, and then as I start going through apache_system_test.sh I can fix more and more of the apache-only config stuff to be general.

@jeffkaufman

This comment has been minimized.

Show comment
Hide comment
@jeffkaufman

jeffkaufman Feb 14, 2013

Contributor

mixes tabs and spaces

mixes tabs and spaces

This comment has been minimized.

Show comment
Hide comment
@oschaaf

oschaaf Feb 14, 2013

Member

fixed in 8083e10

Member

oschaaf replied Feb 14, 2013

fixed in 8083e10

oschaaf and others added some commits Feb 14, 2013

Merge pull request #174 from pagespeed/oschaaf-generate-config-unsupp…
…orted-directives

generate config: don't filter unsupported pagespeed directives in .pyconf files
@jeffkaufman

This comment has been minimized.

Show comment
Hide comment
@jeffkaufman

jeffkaufman Feb 15, 2013

Contributor

Next steps:

  1. I'm going to bring this inside and modify mod_pagespeed's net/instaweb/apache/install/Makefile* to use genconf.

  2. Once that's working, I'll refactor the makefiles to get a big generic chunk and a small apache-specific chunk

  3. Then, back on github, we can write the nginx-specific chunk

And then, with the system tests running, we'll have lots of bugs to look into.

Contributor

jeffkaufman commented Feb 15, 2013

Next steps:

  1. I'm going to bring this inside and modify mod_pagespeed's net/instaweb/apache/install/Makefile* to use genconf.

  2. Once that's working, I'll refactor the makefiles to get a big generic chunk and a small apache-specific chunk

  3. Then, back on github, we can write the nginx-specific chunk

And then, with the system tests running, we'll have lots of bugs to look into.

@jeffkaufman

This comment has been minimized.

Show comment
Hide comment
@jeffkaufman

jeffkaufman Mar 11, 2013

Contributor

When this comes in it's coming via the mod_pagespeed codebase, not via here.

Contributor

jeffkaufman commented Mar 11, 2013

When this comes in it's coming via the mod_pagespeed codebase, not via here.

@jeffkaufman jeffkaufman deleted the oschaaf-generate-config branch Mar 11, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment