Allow master docs to remove from Google search results

[svekars]

We want to unlist this page in Google: https://pytorch.org/docs/master/community/contribution_guide.html - because it goes to the master docs and we want it to go the stable or main docs. The existing way of just unlisting with robots.txt did not give the desired result - the master docs version appeared as a first result in the search. Google suggests to use the nonindex to unlist pages.

I have added the noindex tag to all master docs: pytorch/docs#10

For this change to take effect, we need to allow master docs in robots.txt according to this official Google recommendation:
https://support.google.com/webmasters/answer/7489871?hl=en#zippy=%2Cthis-is-my-site%2Cthe-page-is-blocked-by-robotstxt

At the moment, while the master docs have noindex tag, I still see this as my first result in Google for "PyTorch Contributor Guide":

If you click on Learn Why, it would take you to the aforementioned support.google.com page that suggest to remove the disallow rule from robots.txt.

[netlify]

👷 Deploy Preview for pytorch-dot-org-preview processing.

Name	Link
🔨 Latest commit	e5c5172
🔍 Latest deploy log	https://app.netlify.com/sites/pytorch-dot-org-preview/deploys/66db858471cf290008eaf120

[malfet]

Perhaps I misunderstand something, but changes introduced in pytorch/docs#10 will be overwritten by the next doc build, wouldn't they?

Also, particularly in case of https://pytorch.org/docs/master/community/contribution_guide.html we prefer latest documentation to ones that is mentioned in release docs

[malfet]

Sorry, I misunderstood an intention of this PR, this is to hide docs for deprecated master branch as PyTorch/docs have migrated to main as default branch name.

In that case, would a symlink from master to main be a better solution?

[svekars]

We will create a symlink instead as suggested