Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add robots.txt and a nofollow robots meta tag to dissuade legitimate crawlers from crawling tor2webb-ed instances #6292

Closed
Tracked by #6290
zenmonkeykstop opened this issue Feb 18, 2022 · 0 comments · Fixed by #6299
Closed
Tracked by #6290

add robots.txt and a nofollow robots meta tag to dissuade legitimate crawlers from crawling tor2webb-ed instances #6292

zenmonkeykstop opened this issue Feb 18, 2022 · 0 comments · Fixed by #6299
Assignees
@legoktm

Comments

@zenmonkeykstop
Copy link
Contributor

No description provided.

@zenmonkeykstop zenmonkeykstop mentioned this issue Feb 18, 2022
Closed
4 tasks
@legoktm legoktm self-assigned this Feb 18, 2022
legoktm added a commit that referenced this issue Feb 18, 2022
@legoktm
Ask search engines not to index Source Interface
1784415 
We don't want the SI ending up in traditional clearnet search engines via
tor2web proxies. So add a restrictive robots.txt to disallow all crawlers
that respect it, plus a `<meta name="robots" content="noindex,nofollow">`
tag which tells any search engine that ignores the robots.txt to not index
the page, nor follow any links on the page.

Fixes #6292.
@legoktm legoktm mentioned this issue Feb 18, 2022
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@legoktm legoktm

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Ask search engines not to index Source Interface freedomofpress/securedrop
2 participants
@legoktm @zenmonkeykstop