robots.txt edits

[dchukhin]

https://tree.taiga.io/project/viya-mdn-durable-team/task/59

This pull request follows up on the request make in https://docs.google.com/spreadsheets/d/1X-YLmIg8vVvDWlShLF-361Cz2KcUBnlaPXBHLMaYEaQ/edit#gid=0 to add several more lines to robots.txt.
@stephaniehobson

[codecov-io]

Current coverage is 85.83%

Merging #3884 into master will not change coverage

@@             master      #3884   diff @@
==========================================
  Files           144        144          
  Lines          8566       8566          
  Methods           0          0          
  Messages          0          0          
  Branches       1136       1136          
==========================================
  Hits           7353       7353          
  Misses          977        977          
  Partials        236        236          

Powered by Codecov. Last updated by f50bffe...9b6bdda

[stephaniehobson]

This rule needs to be more precise as it will also block articles like https://developer.mozilla.org/en-US/Firefox/Releases/2/Adding_feed_readers_to_Firefox

[dchukhin]

good 👀

[stephaniehobson]

Thanks for putting it in alphabetical order! A few rules need to be made more precise.

[stephaniehobson]

Need to block everything under users according to the spreadsheet.

[dchukhin]

@stephaniehobson thanks for the review; should be ready for re-review

[stephaniehobson]

This works well for our regular documents and is an improvement over what we have so I'm going to merge. 👍

When I compared some of the test URLs to the file though I discovered that all the $ending urls are not blocked if they're under a zone (yay, another reason for us to stop using zones).

@dchukhin can you do a bit of research to see if the end of the URL matching syntax I saw mentioned on one blog is well supported and, if so, submit another PR using it with the $endings where appropriate (It doesn't look appropriate for $revisions for example). If it's not well supported and you don't recommend switching just comment here.

Example:

Disallow: /*$history$

Thanks.

[dchukhin]

I have looked around and based on here and here it seems that some crawlers (Google, and a few other major ones) support using a $ for the end of the url. I'm not sure if that's enough for what we want or not, since Google is the most popular, but certainly not the only one. Thoughts? @stephaniehobson

[stephaniehobson]

@dchukhin Thanks for doing that research. Google is what we're most concerned about.

I've done a little more thinking on this and I think we can block zones by making the current rules less specific. Could you please change the rules looking for /*docs*$ending to not have the docs directory in them? I'll send an email about this too because git notifications are easy to miss.