/robots.txt is missing

[onnimonni]

Past Issues Searched

• I have searched open and closed issues to make sure that the bug has not yet been reported

Issue is a Bug Report

• This is a bug report and not a feature request, nor asking for support

Describe the bug

Currently when visiting https://lotusdocs.dev/robots.txt it returns 404 error page.

Expected behavior

In order to improve the SEO it would be helpful for the site to return /robots.txt with following content:

User-agent: *
Disallow:
Sitemap: https://lotusdocs.dev/sitemap.xml

This will allow crawling everything and point the crawlers to the sitemap.xml which is already available.

Source for sitemap not supporting relative urls: https://stackoverflow.com/a/14218476/1337062
Source for allowing everything: https://stackoverflow.com/a/44467157/1337062

Screenshots

No response

Environment

No response

[ManuelLevi]

If you add enableRobotsTXT=true to your hugo.toml, it should work.

You can also change the file by editing layouts/robots.txt.