Hide old documentation from google (robots.txt)

[tonigi]

Google returns a mixture of old and new version of manual pages. Perhaps a robots.txt can be used to hide the old ones.

[GiovanniBussi]

I think robots.txt only works in the root directory, so cannot be done on github pages.

We should make a script that adds <meta name="robots" content="noindex"> to all the html files in the manual. See here.

Should not be difficult.

[GiovanniBussi]

Tried here for hand-generated doc-v2.0.

I will wait a few weeks to see if it works (that is if google plumed distance stops reporting manual for v2.0). In case so, I will find out how to fix the travis-generated manuals (v2.1 and up). In this case it is perhaps better to regenerate the manual on travis including the tag.

[GiovanniBussi]

The fix on v2.0 worked (i.e. google search plumed distance does not show v2.0 doc anymore).

Thus I did the following:

• added <meta name="robots" content="noindex"> to all html files related to old versions
• revoked write permission on old manuals to plumedbot so that they are not involuntarily overwritten by travis
• modified .travis/pushdoc script such that it decides based on branch name whether it is supported (and should be indexed) or unsupported (and should be not indexed).

Changes should be effective in google search within a couple of weeks.