docsite: allow content to be removed from the search index if it matc…

[coury-clark]

This PR will allow us to remove certain content files from the docsite search index if they match a pattern in the URL. We have a requirement from marketing that we remove some docs from our search results, and this seemed relatively straightforward.

Example JSON:

{
  "$comment": "This configures the documentation server used in local development only. See ./doc/dev/documentation.md#previewing-changes-locally.",
  "templates": "_resources/templates",
  "content": ".",
  "baseURLPath": "/",
  "rootURL": "https://docs.sourcegraph.com",
  "assets": "_resources/assets",
  "assetsBaseURLPath": "/assets/",
  "check": {
    "ignoreURLPattern": "(^https?://)|(^#)|(^mailto:(hi|support|security|feedback)@sourcegraph\\.com(\\?|$))|(^chrome://)|(^/@)"
  },
  "search": {
    "skipIndexURLPattern": ".*insights*."
  }
}

I was pretty split on how to implement this, and tried a few things:

• At first I wanted to do something similar to the search engine <meta name="robots"...> HTML tag. Unfortunately, it seems that the HTML that is returned from the markdown parser doesn't include the HTML defined in the template file (obvious in hindsight...), and that seemed like a larger refactor that exceeds the bounds of how much we want to invest in this.
• Next I tried introducing a special tag in the Markdown metadata that would remove it from the index. Not sure if I was just doing something wrong but I couldn't get the template to render, so I moved on to the next simplest thing, and here we are.

[LawnGnome]

This looks fine as it is, but which site is this for? For example, I believe handbook.sourcegraph.com doesn't use the built in search any more. (At the very least, it's fronted by swiftype; I don't know if it then falls back to the search when building its corpus.)

[coury-clark]

@LawnGnome This is for the docsite docs.sourcegraph.com - for the Code Insights GA launch we are being asked to unindex any relevant pages for marketing purposes.

[LawnGnome]

@LawnGnome This is for the docsite docs.sourcegraph.com - for the Code Insights GA launch we are being asked to unindex any relevant pages for marketing purposes.

Cool. 👍, then!