Adding HTTP response headers (X-Robots-Tag)

[JingkangZhang]

Select Topic Area

Question

Body

Hi I'd like one of my pdfs hosted on github pages not to be shown in search engine results, and it seems the way to do it is adding X-Robots-Tag http header to the response. There are some previous discussions but just posting to check again - is there a plan to support this? Please enlighten me with other ways for the pdf to not show up in search results too, but I have done some stackoverflow research and it seems the HTTP header approach works best.

[olivierverwoerd]

Make a robots.txt file in the root of your GitHub.io folder
That should be enough

https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt

[JingkangZhang]

Make a robots.txt file in the root of your GitHub.io folder

That should be enough

https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt

I tried it and it didn't work. I still found my pdf in search results after about a month after the change. The exact issue is discussed here and it looks like I need the headers. https://webmasters.stackexchange.com/questions/14520/how-to-prevent-a-pdf-file-from-being-indexed-by-search-engines/49124#49124

[yoannchaudet]

GitHub Pages does not support customizing HTTP headers today.

Since you are worried about Google specifically, I would suggest checking their documentation: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag.

They have alternatives to using a HTTP header for their crawler. For instance a meta HTML tag. After doing the change you will probably want to request your site to be crawled again sooner.

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬