Router: Add support for templated sub-specifications

[d00rman]

Our configuration file has become too long. Moreover, when something changes in it, we have to be careful to propagate them (correctly) into labs, staging, production and MW-Vagrant, making the (manual) process time-consuming and error-prone. This PR tries to minimise config change impacts by adding parts of the private routes (all usually contained in the config file) to specs/sys/v1. Concretely, Mathoid's and MobileApps' /sys/ routes have been moved there.

RESTBase's private routes usually connect it with a back-end service, whose IP changes depending on the environment it is being run in. Hence, this PR also introduces the Swig templating engine which is used to compile and render the subspecs found under the specs/ directory hierarchy. For each template, the complete service configuration is available (more precisely, services[0].conf as passed by service-runner is used).

[d00rman]

cc @wikimedia/services

[gwicke]

+:100: for the general direction! Yay for getting rid of all that ridiculous duplication. This will also let us get rid of some of the /sys/ entry points altogether.

My main worries are about the interaction of swig templates with our config files. The biggest concern is about escaping: We probably don't want swig's HTML escaping, but do need JS string literal encoding / escaping. The other concern is about possible conflicts between swig's syntax {{ var }}, and our own, very similar syntax. The latter might not matter so much in practice.

I was wondering if we could pass a $conf parameter into our template engine instead. That would avoid the escaping issues & possible syntax conflicts, but might be slightly more complex to implement.

Edit: Another thought might be to localize the configs a bit, so that we can specify the options pertaining to a spec right where it's pulled in, very similar to how we currently do this for modules. This would more clearly define the config interface for each module, and avoid littering the global config with all kinds of module-specific bits. It would also reflect that sub-specs are an alternative to modules.

[d00rman]

We probably don't want swig's HTML escaping,

Correct, and we tell Swig not to do it.

but do need JS string literal encoding / escaping.

Not sure I see why. Templating is applied on start-up once, and variables will appear as specified in the config (in the exact same way that we do it now in prod with Puppet and ERB).

The other concern is about possible conflicts between swig's syntax {{ var }}, and our own, very similar syntax.

That does indeed look similar, but even if do make the mistake of using one instead of the other, it pretty likely the mistake will be made apparent pretty quickly. Alternatively, we might change Swig's delimiters to something like ERB's <% %>, which would give us config consistency with ops/puppet to some extent.

I was wondering if we could pass a $conf parameter into our template engine instead. That would avoid the escaping issues & possible syntax conflicts, but might be slightly more complex to implement.

I was wondering the same, but having static, one-time compilation and rendering seems like a good first step. The benefit here is that the templates are compiled and executed only once on start-up, so the net effect is that you have the exact same spec as before, hence no per-request additional overhead (even though, compared to the network request that is bound to happen after that is negligible).

Another thought might be to localize the configs a bit, so that we can specify the options pertaining to a spec right where it's pulled in, very similar to how we currently do this for modules. This would more clearly define the config interface for each module, and avoid littering the global config with all kinds of module-specific bits. It would also reflect that sub-specs are an alternative to modules.

Good point.

[gwicke]

We probably don't want swig's HTML escaping,

Correct, and we tell Swig not to do it.

Ah, good. Missed that, sorry.

but do need JS string literal encoding / escaping.

Not sure I see why. Templating is applied on start-up once, and variables will appear as specified in the config (in the exact same way that we do it now in prod with Puppet and ERB).

Example:

uri: /some/{$.default($.request.params.foo, $conf.defaultFoo)}

Injecting $conf.defaultFoo with textual substitution ({{ $conf.defaultFoo }}) requires proper escaping / string encoding, so that it works out to a valid string constant in the expression. If we hook into the regular templating engine, we avoid the need for textual substitution.

[d00rman]

Counter-example:

uri: {{ my_service_uri }}/path

Here we wouldn't want to escape chars, since my_service_uri might contain genuine paths. So, I think that escaping is something we'll have to deal with on per-case basis.

[gwicke]

@d00rman, that shouldn't be an issue:

uri: {+ $conf.service_uri }/path

[gwicke]

Another consideration is the ability to pass around more complex sub-structures, like entire objects or arrays. Doing that as strings might get quite ugly.

[gwicke]

@d00rman, I implemented a variant with module-like option support in #415.

[d00rman]

Closing in favour of #415