Support for robots.txt

[pskd73]

Is your feature request related to a problem? Please describe.
robots.txt is an important tool in SEO. I am missing it in my own setup. Better we support it out of the box

I am happy to contribute if it is not implemented yet :)

[sjwaight]

A simple workaround is to create a static 'robots.txt' file in the public folder and it will be deployed to your target hosting environment.

[bravo-kernel]

@sjwaight I have motivated my suggestion for using next-sitemap in more detail at timlrx/pliny#101.

If a PR is accepted I think that would be the correct way forward.

@pskd73 implementing is quite easy, see bravo-kernel/bravo-kernel.com#169

[timlrx]

v2 which is based on app dir (coming soon) will allow you to add a robots.txt directly. I might just add it in the template as well.