Configurable robots.txt

[abovedave]

If you visit the CDN url /robots.txt you get an error:

{"statusCode":404,"detail":"'robots.txt' is not a valid route, recipe, processor or image format"}

Would be handy to be able to configure a correct response for indexing robots.

[jimlambie]

@abovebored when you say "a configurable response", do you mean just the ability to return a robots.txt file from the root or a public folder?

[abovedave]

@jimlambie yep, so choosing who you disallow etc

[jimlambie]

@abovebored please pull branch 1.7.0-rc.1 and test this? see config details in https://github.com/dadi/cdn/blob/1.7.0-rc.1/CHANGELOG.md#170---2016-11-30