Is it possible to have permissive default?

[evgenyfadeev]

We've installed the app on our production site where we did not want to disallow any crawling.

Unfortunately - with the disallow all default we had an issue with Google, which basically stopped all search traffic just upon upgrading the application. We've lost 60% of traffic instantly.

[beniwohli]

+1. While I agree that usually "secure by default" is the better choice, I think in this case an "allow all" default is less surprising.

[jezdez]

This would be a rather annoying change for all current users, but I see your argument. I'll change the default in the next upcoming major release (1.0).

[merenbach]

I have an issue that I suspect may be related. I am trying to create a permissive setting manually and I am not having much luck. I want to specify:

User-agent: *
Disallow: 

I can't seem to do this; if I put in just a space character in the url definition, it becomes a slash. I thought about using Allow: /, but I don't know if the Allow directive is quite as widely supported as Disallow.

If this should have been in a separate issue, please let me know. Or maybe I'm going about this the wrong way. ;)

Side note: In this vein, I noticed that the docs say this:

The sites framework is used to enable multiple robots.txt per Django instance. If no rule exists it automatically allows every web robot access to every URL.

[kenzic]

+1 x 1,000,000

[jezdez]

Okay, I haven't gotten to this in the last months, so if there is interest I accept pull requests.