Add robots.txt instructions

[alecslupu]

🎩 What? Why?

In this PR we add some robots.txt ruleset, that would prevent search engine crawling certain pages from the platform ( search and profiles). as requested here

📌 Related Issues

Link your PR to an issue

• Related to EPIC Spam #10039
• Related to Add meta robots noindex to search and profile  #10120

Testing

1. Make sure you run the bundle exec rails decidim:robots:replace
2. Check your development_app robots.txt for the following lines

# the following adds a rule for all bots to not index any page that contains a profile or search
User-agent: *
Disallow: /profiles/
Disallow: /search

3. Generate a new application & check the generated robots.txt file

📷 Screenshots

♥️ Thank you!

[andreslucena]

As this is already done in develop with #10120, what I had in mind is to do this directly in v0.27 (release/0.27-stable base branch) and backport to v0.26. Can you change the base branch? You probably need to start a new PR, I'm not too sure on how GH works with these kind of changes.

Also I was thinking in putting the instructions manually, but having the rake task is 👌🏽.

[alecslupu]

As this is already done in develop with #10120, what I had in mind is to do this directly in v0.27 (release/0.27-stable base branch) and backport to v0.26. Can you change the base branch? You probably need to start a new PR, I'm not too sure on how GH works with these kind of changes.

Also I was thinking in putting the instructions manually, but having the rake task is 👌🏽.

Actually this is complementary to #10120 .

• A robots.txt file controls crawling. It instructs robots (a.k.a. spiders) that are looking for pages to crawl to “keep out” of certain places. You place this file in your website’s root directory.
• A noindex tag controls indexing. It tells spiders that the page should not be indexed. You place this tag in the code of the relevant web page. Here is an example of the tag:

In other words, if we just keep the meta tag, the search engines may still crawl those pages. Using robots.txt we tell the search engines that /profile or /search are "off limits"

[andreslucena]

In other words, if we just keep the meta tag, the search engines may still crawl those pages. Using robots.txt we tell the search engines that /profile or /search are "off limits"

I have to admit that in my head these were interchangeable, and the main difference was if you had direct access to the application/HTML/views vs being able to just add/change a TXT file in the server. 🤷🏽

In our case, I think that both of these cases are good: this is more "hard" against the anti-spam measures, but as its actually a manual task that need to be run, and it could be missed by implementers when updating, the change from #10120 is like a safety net just in case.

Thanks for the explanation!

[andreslucena]

I have some suggestions before merging this one

[alecslupu]

@andreslucena let me know if there are any improvements to be done, or is a good to go.

[andreslucena]

LGTM, I'll accept the changes so we don't break the generator CI and we can merge this bad boy

[andreslucena]

Merging even though we have some failing specs, as they're expected

[ahukkanen]

@alecslupu Is is by design that every time I want to create a test app or development app, I get this conflict that I need to choose an option for:

    conflict  public/robots.txt
Overwrite /.../decidim/spec/decidim_dummy_app/public/robots.txt? (enter "h" for help) [Ynaqdhm] 
       force  public/robots.txt

We can also see this same in the CI runs:

[alecslupu]

@alecslupu Is is by design that every time I want to create a test app or development app, I get this conflict that I need to choose an option for:

    conflict  public/robots.txt
Overwrite /.../decidim/spec/decidim_dummy_app/public/robots.txt? (enter "h" for help) [Ynaqdhm] 
       force  public/robots.txt

We can also see this same in the CI runs:

@ahukkanen this was initially designed to be appended to the existing file, but @andreslucena requested something else.

i can patch it if @andreslucena agrees.

Please see for additional details #11693 (comment)

[ahukkanen]

@ahukkanen this was initially designed to be appended to the existing file, but @andreslucena requested something else.

i can patch it if @andreslucena agrees.

Please see for additional details #11693 (comment)

OK, I personally find it cumbersome that I have to manually agree on the rewrite of this file during the application generation.

It makes total sense when we are modifying an existing application to have the user to confirm the change but when generating new applications, it just adds one extra step. Since it takes a while to generate the application, I generally start it and go to get coffee or something while I'm waiting for the application to be generated. Now I had to find out that it wasn't actually generated because it was stuck waiting for user input.

[andreslucena]

It makes total sense when we are modifying an existing application to have the user to confirm the change but when generating new applications, it just adds one extra step.

I agree, that's a bug. For new apps this should be override without asking for confirmation. I didn't check that out until today and didn't have time to make the PR yet.

[andreslucena]

Just to be clear:

• For existing apps, ask for confirmation (as it's a possible destructive action)
• For new apps, do it without confirmation