Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Robots file missing #71

Closed
danielcompton opened this issue Feb 7, 2016 · 2 comments
Closed

Robots file missing #71

danielcompton opened this issue Feb 7, 2016 · 2 comments

Comments

@danielcompton
Copy link
Contributor

It looks like www.clojure.org/robots.txt is 404'ing, and there used to be a sitemap for clojure.org which is also missing.
@puredanger
Copy link
Member

http://clojure.org/sitemap.xml is there.

As this is a static site, we don't really have any pages to disallow for robots. Is there any reason to have a robots.txt?

@danielcompton
Copy link
Contributor Author

Ah, missed the sitemap.

It's considered good practice to exclude search results like http://clojure.org/search?q=test from indexing. Also, you can link to the sitemap from your robots.txt file which saves you from submitting it to many different search engines - http://www.sitemaps.org/protocol.html#submit_robots

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@puredanger @danielcompton