Make it easier to disallow robots crawling. #53

rik opened this Issue Jul 27, 2011 · 5 comments


None yet
4 participants

rik was assigned Jul 27, 2011


fwenzel commented Jul 27, 2011

Sounds good! Rik, can you whip this into shape for Playdoh? The code itself should probably live in funfactory, and playdoh itself should just point to it. You might also not have to render a full-blown view like AMO does:


fwenzel commented Sep 29, 2011

@kumar303 can you advise where in playdoh/funfactory Anthony should put this? Thanks!


kumar303 commented Sep 29, 2011

Oh yeah, that should definitely be in playdoh. AMO checks the services domain so I think we can strip that out and just check ENGAGE_ROBOTS. Developers can set ENGAGE_ROBOTS=True on the public site and keep it False everywhere else.

Something like this?

diff --git a/project/settings/ b/project/settings/
index 34c388e..cce6f85 100644
--- a/project/settings/
+++ b/project/settings/

+# Should robots.txt deny everything or disallow a calculated list of URLs we
+# don't want to be crawled? Default is false, disallow everything.
+# Also see

Always generate a CSRF token for anonymous users.


diff --git a/project/ b/project/
index afe33c2..f71b617 100644
--- a/project/
+++ b/project/
@@ -14,6 +14,14 @@ patch()
urlpatterns = patterns('',
# Example:
(r'', include(urls)),

  • Generate a robots.txt

  • (r'^robots.txt$',
  •    lambda r: HttpResponse(
  •        "User-agent: *\n%s: /" % 'Allow' if settings.ENGAGE_ROBOTS else 'Disallow' ,
  •        mimetype="text/plain"
  •    )
  • )

Uncomment the admin/doc line below to enable admin documentation:

(r'^admin/doc/', include('django.contrib.admindocs.urls')),


kumar303 commented Jul 16, 2012

That looks good to me. It should also be set to True in settings/ with a comment about it. The reason for that is IT likes the committed version of to as close to production as possible. This change might break the funfactory installer because it uses patch on settings files. You can try installing an app after applying the change locally to see.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment