Removed Disallow: /cache/

[pe7er]

This PR is to fix #6939
I removed "Disallow: /cache/" so that Google will be able to index CSS & JS files stored in /cache/

[810]

@test
+1

[Fedik]

well, this is controversial issue, as Joomla store css/js under /media and under /templates

I just can say, if you worried about Google, then you can use "Allow: /cache/blablabla" in your site installation .... it not really standard but Google allow this

[mbabker]

My concern with removing the cache folder is it is used by devs for more than web assets. I know my own extensions and some Akeeba products store data there that Google shouldn't be indexing.

[Casporro]

I have a yootheme template and the minify and gzip compresion store the files inside the cache folder... Because of that Google penalize the page... Is a mistake of the template use this folder? Could be a problem remove the disallow cache folder?

[mbabker]

It isn't an issue that developers use the cache folder for stuff, it's Joomla's system-wide cache folder. The only concern is the added emphasis on mobile readiness and for Google that means being able to index all web assets. If the cache folder only stored that kind of stuff, there'd be no issue removing it from the disallow list.

But, the folder is used for more than that. HTML files get written here with cached output when Joomla's system cache is enabled and extensions cache data here (be it update information, cached data from external services, or web assets). So it isn't a folder that we should completely allow search engines to index.

I'd be concerned if Google were to start indexing site cache folders and exposing that cached data that shouldn't be directly accessed, which is why I'm making these comments. Removing the disallow on the cache folder isn't as clear of a decision IMO as removing it on the images, media, or templates folders.

[Fedik]

@Casporro I think template/extension developers should warn user about such behavior, so they can decide allow or disallow ...
but make it global is not very good idea, @mbabker already explained the reason

[Casporro]

Thanks @mbabker and @Fedik, I will add disallow for the subfolders of cache folder I wan't google see. Is it a good solution in your opinion?

[Fedik]

@Casporro I would suggest add Allow, but only for folders that you want to allow, like google do

Disallow: /cache
Allow: /cache/folder-to-allow

I also use such thing on my sites 😄

[Casporro]

Thank you @Fedik
I'll do the same!

[brianteeman]

I really dont agree with this
the cache should not be searched
A user can always change it for their own site if they want to

On 14 May 2015 at 14:59, Casporro notifications@github.com wrote:

Thank you @Fedik https://github.com/Fedik
I'll do the same!

—
Reply to this email directly or view it on GitHub
#6940 (comment).

Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
http://brian.teeman.net/

[Casporro]

@brianteeman And what do you propose to Google not penalize you in these cases?
I don't want to move from the first page to page 86 of the results for this reason, when the page is responsive even Google thinks it isn't. Unfortunately they have the power to do whatever they want, because it's the most used search engine.

[brianteeman]

You edit the robots file. Problem solved.

On 14 May 2015 at 19:51, Casporro notifications@github.com wrote:

@brianteeman https://github.com/brianteeman And what do you propose to
Google not penalize you in these cases?
I don't want to move from the first page to page 86 of the results for
this reason, when the page is responsive even Google thinks it isn't.
Unfortunately they have the power to do whatever they want, because it's
the most used search engine.

—
Reply to this email directly or view it on GitHub
#6940 (comment).

Brian Teeman
Co-founder Joomla! and OpenSourceMatters Inc.
http://brian.teeman.net/

[ghost]

@brianteeman 👍

[Casporro]

@brianteeman I don't agree with remove Disallow: /cache/ too, but I think it could be at least a warning or an easier solution from the administration panel.

[brianteeman]

Setting to needs review so the CMS maintainers can make a decision.

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/6940.}

[Bakual]

Based on the various comments I'm rejecting this PR.
I also agree that Google shouldn't index the cache folder by default.