Add 'noindex' directive for web crawlers to basic theme search results

[jayaddison]

Feature or Bugfix

• Bugfix

Purpose

• Web crawlers should generally prefer to crawl the contents of the documentation and build their own index of those; the search.html page implements client-side search, so there's no value to web crawlers in indexing it, and we should communicate to them that they avoid doing that.

Detail

• Add an HTML meta element robots with noindex configured.
• See also:
  • https://en.wikipedia.org/wiki/Noindex
  • https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag
  • https://moz.com/learn/seo/robots-meta-directives

Relates

• Resolves HTML search should set noindex #11697.

Edit: update the description to match my understanding that all search results in search.html are from client-side searches.

[AA-Turner]

Could we add a CHANGES entry? I think tests can be skipped if you want, though.

A