Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Is it possible to have permissive default? #10

Closed
evgenyfadeev opened this Issue · 5 comments

5 participants

@evgenyfadeev

We've installed the app on our production site where we did not want to disallow any crawling.

Unfortunately - with the disallow all default we had an issue with Google, which basically stopped all search traffic just upon upgrading the application. We've lost 60% of traffic instantly.

@piquadrat

+1. While I agree that usually "secure by default" is the better choice, I think in this case an "allow all" default is less surprising.

@jezdez
Owner

This would be a rather annoying change for all current users, but I see your argument. I'll change the default in the next upcoming major release (1.0).

@merenbach

I have an issue that I suspect may be related. I am trying to create a permissive setting manually and I am not having much luck. I want to specify:

User-agent: *
Disallow: 

I can't seem to do this; if I put in just a space character in the url definition, it becomes a slash. I thought about using Allow: /, but I don't know if the Allow directive is quite as widely supported as Disallow.

If this should have been in a separate issue, please let me know. Or maybe I'm going about this the wrong way. ;)

Side note: In this vein, I noticed that the docs say this:

The sites framework is used to enable multiple robots.txt per Django instance. If no rule exists it automatically allows every web robot access to every URL.

@kenzic

+1 x 1,000,000

@jezdez
Owner

Okay, I haven't gotten to this in the last months, so if there is interest I accept pull requests.

@gbezyuk gbezyuk referenced this issue from a commit in gbezyuk/django-robots
@gbezyuk gbezyuk fix issue #10 - allow: / by default 5eedd6f
@jezdez jezdez closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.