Serve robots.txt to prevent crawlers from accessing UI

[brandur]

Here, add a robots.txt that denies all crawler access to UI
installations. If a River UI is exposed publicly without authentication,
it's bad and a security problem, but we don't have to make it worse by
allowing crawlers to find it.

We'll add a separate robots.txt in the demo that will allow basic
access to top-level pages, although deny access to crawl through the
potentially tens of thousands of jobs in /jobs/.

There were a number of ways to go about this that were plausible. I
ended up putting in an embedded file system that can pull in static
files easily into the Go program, and which could be reused for future
static files in case they're needed. It may be a little more than we
need right now, but it wasn't much harder to do than serving a one off
static file.

[bgentry]

Do you think this makes sense in the generalized riverui handler, or should it be specific to the demo app? Open to either, just not sure if this is something regular users will want (or if it makes sense path prefixed within the handler).

[brandur]

Thx.

Was going to mention too that I feel like what I ended up with here is a bit of overkill in that it allows generic files to be added along with robots.txt, but in the end it wasn't that much less code than just having an endpoint for a single file only. I figure we can leave it like this and then refactor at some point in the future if it turns out that most of this code has gone relatively unused.