robots.txt in github.io pages

[williamadba]

Select Topic Area

Question

Body

Do crawlers honor a robots.txt page in the root of a GitHub.io folder? Any documentation or guidance on this?

[arafajs]

Yes, crawlers will honor a robots.txt page in the root of a GitHub.io folder.
GitHub does not provide any official documentation specific to using robots.txt on GitHub Pages. But the standard robots.txt syntax and conventions still apply.

[arafajs]

Documentation:
https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt

[williamadba]

Thanks. Was concerned about github.io pages and robots.txt specifically, this helps.