don't register stackdriver metrics on every instance start

[mikehelmick]

TL;DR

Registering on every instance start causes us to get throttled on the stackdriver API and can cause errors on startup.

[sethvargo]

I think there's a few ways we could do this:

1. Move the metrics registration process into its own binary that we execute as part of the build/deploy steps. This ensures it happens once per deploy (which is what we really want). I think I prefer this approach.

2. Move the metrics creation back into Terraform. It started this way, but became unsustainable to keep them in sync. Now that metrics creation is rare, we could move it back. It would be ugly, since we'd have to terraform import all the existing metrics.

3. Establish a lower-level connection to Redis and write the current build-id somewhere. If no value exists, do the registration. This has some complex edge cases. It makes connecting to Redis a SPOF for startups, but we already have that. Another challenge is that we can't use a nice client here, because we don't establish the client until after metrics have been registered (since the client emits metrics).

Other ideas?

[mikehelmick]

I was thinking something along the lines of 1

Separate binary, but deployed as a service.
So when it's deployed, and the container starts, the metrics will register.

[sethvargo]

Ah - and then set min/max instances to 1?

[sethvargo]

#1549