Generate configuration files for apache and nginx

[oschaaf]

This pull implements generating apache and nginx configuration files for testing purposes

You can generate an nginx configuration file using:

./cli.py pagespeed.debug.pyconf nginx [directive] > test.nginx.conf

This can be tested using, for example:

/usr/local/nginx/sbin/nginx -t -c <path_to>/test.nginx.conf

An apache configuration file can be generated using

./cli.py pagespeed.debug.pyconf apache [directive] > test.apache.conf

[jeffkaufman]

Is this actually a problem? Do we need the #ifndef ALL_DIRECTIVES everywhere? They make it really hard to read. What happens if we just include the directive twice? Or move ALL_DIRECTIVES into its own vhost or something? Or even wrap everything in an giant #ifndef ALL_DIRECTIVES except and then a separate #ifdef ALL_DIRECTIVES at the end?

[oschaaf]

@jeffkaufman
Currently, including the a directive with the same name twice will result in an error noting that it is already defined ( an array of values can be used to specify multiple values for the same directive instead).

Moving the ALL_DIRECTIVES to a separate vhost sounds like a good idea to me, do you want me to do that?

[jeffkaufman]

To test the config system I ran:

for server in apache nginx ; do for mode in ALL_DIRECTIVES COVERAGE FURIOUS_GA FURIOUS_NO_GA GZIP HTTPS MEMCACHE_COVERAGE MEMCACHED PER_VHOST_STATS PROXY REWRITE SHARED_MEM_LOCKS SLURP SPELING STATS_LOGGING STRESS ; do CMD="python cli.py pagespeed.debug.pyconf $server $mode" ; $CMD > $server.$mode.conf || echo $CMD ; done ; done

This turned up some simple errors, which I fixed: #169

Two errors remain which weren't easily fixed:

$ python cli.py pagespeed.debug.pyconf nginx PER_VHOST_STATS
Traceback (most recent call last):
  File "cli.py", line 81, in <module>
    placeholders, template)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/genconf.py", line 244, in execute_template
    text = template.render(config=config)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/templite.py", line 81, in render
    eval(self.__code, namespace)
  File "<templite '${\n# Copyright 2013 '>", line 222, in <module>
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 38, in write_cfg
    write_cfg(key_to_writer, item, level + 1)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 38, in write_cfg
    write_cfg(key_to_writer, item, level + 1)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 51, in write_cfg
    raise Error("no tranform handler for [%s]" % key)
util.Error: no tranform handler for [literal]

$ python cli.py pagespeed.debug.pyconf nginx STRESS
Traceback (most recent call last):
  File "cli.py", line 81, in <module>
    placeholders, template)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/genconf.py", line 244, in execute_template
    text = template.render(config=config)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/templite.py", line 81, in render
    eval(self.__code, namespace)
  File "<templite '${\n# Copyright 2013 '>", line 222, in <module>
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 38, in write_cfg
    write_cfg(key_to_writer, item, level + 1)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 38, in write_cfg
    write_cfg(key_to_writer, item, level + 1)
  File "/home/jefftk/ngx_ps/ngx_pagespeed/test/genconf/util.py", line 30, in write_cfg
    handled = w(config[key], level)
  File "<templite '${\n# Copyright 2013 '>", line 168, in write_add_options_open
Exception: Option 'ExecCGI': no translation for nginx

It would be nice if errors could be accompanied by the line number the error's on.

[jeffkaufman]

Moving the ALL_DIRECTIVES to a separate vhost sounds like a good idea to me, do you want me to do that?

That would be great. Or, if you'd rather, I can do it.

[oschaaf]

@jeffkaufman
"It would be nice if errors could be accompanied by the line number the error's on."

That should be very useful, but it's a little harder then I thought initially, because we have semi-inheritance for locations and header directives in .pyconf.
To be able to make a best effort to display line numbers, the script needs to somehow track the source of these properties as they get inherited. I am looking in to that.

[jeffkaufman]

This is apache-only

[oschaaf]

@jeffkaufman: "This is apache-only"
Do you think it would be feasible to have something like http://wiki.nginx.org/SimpleCGI generated for nginx at this point?

[jeffkaufman]

Probably. For now I'd be inclined to mark it as apache only, and then as I start going through apache_system_test.sh I can fix more and more of the apache-only config stuff to be general.

[jeffkaufman]

Next steps:

1. I'm going to bring this inside and modify mod_pagespeed's net/instaweb/apache/install/Makefile* to use genconf.

2. Once that's working, I'll refactor the makefiles to get a big generic chunk and a small apache-specific chunk

3. Then, back on github, we can write the nginx-specific chunk

And then, with the system tests running, we'll have lots of bugs to look into.

[jeffkaufman]

When this comes in it's coming via the mod_pagespeed codebase, not via here.