Robots.txt auto generation on install

[macik]

Thus sitemap directive added into robots.txt (see 7523f28 ) we have ability to autocreate (if write permition granted) robots.txt with certain site url during install. So no addition (hand made) changes required to robots.txt (except uncomment sitemap line).

[seditio]

Good point. But a more important thing is the Host directive.

[Dayver]

But only Yandex

[macik]

Yes its Yandex related feature, not supported by others.

[macik]

But we can use somekind of this variant:

# Default Cototnti exclusions
User-agent: *
Disallow: /datas
Disallow: /images
Disallow: /js
Disallow: /lang
Disallow: /lib
Disallow: /themes
Disallow: /system

# Yandex related directives
User-agent: Yandex
Disallow: /datas
Disallow: /images
Disallow: /js
Disallow: /lang
Disallow: /lib
Disallow: /themes
Disallow: /system
Host: your-domain.com

# For sitemaps.xml autodiscovery. Uncomment if you have one:
# Sitemap: http://your-domain.com/sitemap.xml

[trustmaster]

This requires setting a proper CHMOD on robots.txt file which isn't very good for 2 reasons: it adds one more step to the installation process and it isn't very good for security.

[macik]

No special actions needed. Just one string help these people had writable root by default (eg. on sharing hosting it's a common practice).
@file_put_contents("\n./.htaccess", array('# Sitemap: '.$cfg['mainurl']."\n"),FILE_APPEND);

[Kilandor]

Such a limited scope of things needing to modify robots.txt I would instead propose that simply on installing sitemap plugin. The attempt to modify robots.txt can be made during the installation of the plugin, the plugin handling it itself. It could also check and warn about modification if need be.

[macik]

Good point, thanks.

[Alex300]

Added automatic creation of directives Sitemap and Host