robots.txt

[vat9]

hugo v0.16-DEV-43B5DFA does not preserve static/robots.txt. Instead we get the default:

User-agent: *

in public/robots.txt

[bep]

/cc @ifraixedes

[ifraixedes]

I've checked it and behaves @vat9 mentioned.

Be aware that I'm not one of the maintainers of the project, so my comment may not be exactly right.

My thoughts is that robots.txt is treated as any other template, so if I do the same with any other template file it happens the same. For example I remove layouts/index.html and I place index.html in static (static/index.html) the public/index.html is the one generated by hugo, no the one placed in static.

Due that, I'm not sure if it could be considered a bug.

The solution, at least for now, it's to place robots.txt in layouts/robots.txt; if the file doesn't have any template lang, it will be copied as it is from there to public

[vat9]

I've toggled the param disableRobotsTXT as well and still end up with the same public/robots.txt file. Placing it in layouts is a work around--or my concept of what is static and what is used for template generation may just be off.

[bep]

@vat9 I have reverted the flag logic so robots.txt isn't created as a default. But disableRobotsTXT should also work, I just tested it.

[bep]

Fixed in 54750b0

[ifraixedes]

@vat9 I forgot the existence of that flag.
@bep great thanks, probably that logic make more sense, when I added it, I applied the same logic of the code around without thinking that much.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.