New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add current robots.txt content in controller #6523

Merged
merged 3 commits into from Oct 20, 2016

Conversation

Projects
None yet
4 participants
@Quetzacoalt91
Member

Quetzacoalt91 commented Sep 28, 2016

Questions Answers
Branch? develop
Description? We now have a default robots.txt in the PrestaShop project, which is nice. However, every time we regenerate it, we lose all its content because the content has not been copied in the controller.
Type? bug fix
Category? CO
BC breaks? Nope
Deprecations? Nope
Fixed ticket? /
How to test? Regenerate the file robots.txt in the Traffic > SEO & URLs tabs, and check you still have the default folders of #5876.
Show outdated Hide outdated controllers/admin/AdminMetaController.php Outdated
@aleeks

This comment has been minimized.

Show comment
Hide comment
@aleeks
Contributor

aleeks commented Oct 8, 2016

@Quetzacoalt91

This comment has been minimized.

Show comment
Hide comment
@Quetzacoalt91

Quetzacoalt91 Oct 10, 2016

Member

Hoy,

FYI, this array has been updated regarding the previously accepted robots.txt.

But your questions are legitimate. if I remember well the folders your talked about were existing at that time. I'm pretty sure this was just before we add Symfony to the project.

modules needs to be allowed, it can contains interesting assets for the robots. test would disppear on each PrestaShop and won't be available on production.

The controller AND the file robots.txt will be updated.

Member

Quetzacoalt91 commented Oct 10, 2016

Hoy,

FYI, this array has been updated regarding the previously accepted robots.txt.

But your questions are legitimate. if I remember well the folders your talked about were existing at that time. I'm pretty sure this was just before we add Symfony to the project.

modules needs to be allowed, it can contains interesting assets for the robots. test would disppear on each PrestaShop and won't be available on production.

The controller AND the file robots.txt will be updated.

@julienbourdeau

This comment has been minimized.

Show comment
Hide comment
@julienbourdeau

julienbourdeau Oct 10, 2016

Contributor

I also think img/ should be indexed (I answer based on the previous comments, I didn't test the PR) ^^

Contributor

julienbourdeau commented Oct 10, 2016

I also think img/ should be indexed (I answer based on the previous comments, I didn't test the PR) ^^

@Quetzacoalt91

This comment has been minimized.

Show comment
Hide comment
@Quetzacoalt91

Quetzacoalt91 Oct 10, 2016

Member

PR Updated

Member

Quetzacoalt91 commented Oct 10, 2016

PR Updated

@aleeks

More, I generate the file, and I think we have some problems :
Disallow: /*fr/adresses
Disallow: /*fr/adresses
Disallow: /*fr/connexion
Disallow: /*fr/connexion
Line are duplicated

On the sql Line 764, we need to : SELECT DISTINCT ml.url_rewrite, l.iso_code,
Without :
0 =>
array (size=2)
'url_rewrite' => string 'adresse' (length=7)
'iso_code' => string 'fr' (length=2)
1 =>
array (size=2)
'url_rewrite' => string 'adresse' (length=7)
'iso_code' => string 'fr' (length=2)
2 =>
array (size=2)
'url_rewrite' => string 'adresse' (length=7)
'iso_code' => string 'en' (length=2)
3 =>
array (size=2)
'url_rewrite' => string 'adresse' (length=7)
'iso_code' => string 'en' (length=2)

With :
0 =>
array (size=2)
'url_rewrite' => string 'adresse' (length=7)
'iso_code' => string 'fr' (length=2)
1 =>
array (size=2)
'url_rewrite' => string 'adresse' (length=7)
'iso_code' => string 'en' (length=2)

if we have multiple shop, link are duplicated. (because of id_shop)

@@ -746,7 +746,9 @@ public function getRobotsContent()
);
// Directories
$tab['Directories'] = array('classes/', 'config/', 'download/', 'mails/', 'modules/', 'translations/', 'tools/');
$tab['Directories'] = array('cache/', 'classes/', 'config/', 'controllers/',

This comment has been minimized.

@aleeks

aleeks Oct 17, 2016

Contributor

After comparing with 1.6, /modules/ was put in the file because on top, we have :
Allow: */modules/*.css
Allow: */modules/*.js
Allow: */modules/*.png
Allow: */modules/*.jpg
so we can keep modules ?

@aleeks

aleeks Oct 17, 2016

Contributor

After comparing with 1.6, /modules/ was put in the file because on top, we have :
Allow: */modules/*.css
Allow: */modules/*.js
Allow: */modules/*.png
Allow: */modules/*.jpg
so we can keep modules ?

@aleeks

aleeks approved these changes Oct 20, 2016

@@ -484,7 +484,7 @@ public function generateRobotsFile()
fwrite($write_fd, "# Files\n");
foreach ($this->rb_data['Files'] as $iso_code => $files) {
foreach ($files as $file) {
if (!empty($language_ids)) {
if (!empty($language_ids) && count($language_ids) > 1) {

This comment has been minimized.

@aleeks

aleeks Oct 20, 2016

Contributor

It's for multilanguage.

Multi :
-Disallow: /_fr/recuperation-mot-de-passe
-Disallow: /_fr/recherche
-Disallow: /_en/password
-Disallow: /_en/search

Not multi :
Disallow: /panier
Disallow: /recherche

@aleeks

aleeks Oct 20, 2016

Contributor

It's for multilanguage.

Multi :
-Disallow: /_fr/recuperation-mot-de-passe
-Disallow: /_fr/recherche
-Disallow: /_en/password
-Disallow: /_en/search

Not multi :
Disallow: /panier
Disallow: /recherche

@@ -761,7 +761,7 @@ public function getRobotsContent()
// Rewrite files
$tab['Files'] = array();
if (Configuration::get('PS_REWRITING_SETTINGS')) {
$sql = 'SELECT ml.url_rewrite, l.iso_code
$sql = 'SELECT DISTINCT ml.url_rewrite, l.iso_code

This comment has been minimized.

@aleeks

aleeks Oct 20, 2016

Contributor

It's for not duplicate line (multi shop trouble!)

@aleeks

aleeks Oct 20, 2016

Contributor

It's for not duplicate line (multi shop trouble!)

@maximebiloe maximebiloe merged commit ccb3025 into PrestaShop:develop Oct 20, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@maximebiloe maximebiloe deleted the Quetzacoalt91:PR5876-ctrl-update branch Oct 20, 2016

@maximebiloe

This comment has been minimized.

Show comment
Hide comment
@maximebiloe

maximebiloe Oct 20, 2016

Contributor

Thank you!

Contributor

maximebiloe commented Oct 20, 2016

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment