Metrics: private -> internal & <default> -> external

[gwicke]

• Rename 'private' metrics to 'internal'.
• Prefix external metrics with 'external', rather than the empty default. This
  makes this symmetric with internal / internal-update requests, which in turn
  simplifies wildcard selections in dashboards.

Note: Deploying this involves several metric renames, and should be
coordinated with ops to avoid losing metrics.

[gwicke]

/cc @wikimedia/services

[d00rman]

@filippog @eevans this is the same as wikimedia/restbase#498 . Could you two coordinate on deploying this and renaming the metrics early next week?

[eevans]

@filippog @eevans this is the same as wikimedia/restbase#498 . Could you two coordinate on deploying this and renaming the metrics early next week?

@d00rman Sure, I can take it.

@gwicke can we get a version bump?

[eevans]

Bump? Can we get a rebase and version increase?

[gwicke]

Rebased on top of @Pchelolo's changes. I omitted the version bump to avoid conflicts with the bump for wikimedia/swagger-router#38, which we plan to deploy today.

[gwicke]

@filippog @eevans: Deploy PR is at https://gerrit.wikimedia.org/r/#/c/271194/.

The following metric renames are needed:

restbase.private -> restbase.internal
restbase.v1_* -> restbase.external.
restbase.sys_* -> restbase.external.
restbase.{ALL,GET,HEAD,OPTIONS,POST,_robots} -> restbase.external.

The following metric hierarchies are old and can be removed:

restbase.ab
restbase.wikimedia

[d00rman]

I also prepared the ansible PR - wikimedia/ansible-deploy#47 . Just merge it when the time comes :)

[filippog]

deployed yesterday, still seeing some traffic for some sys_ metrics, e.g.

10:08:18.379655 IP restbase1002.eqiad.wmnet.39077 > graphite1001.eqiad.wmnet.8125: UDP, length 555
E..G"2@.?..|
@..
@ ......3Nxrestbase.sys_post-data_-bucket-.PUT.ALL:0|ms

[eevans]

deployed yesterday, still seeing some traffic for some sys_ metrics, e.g.

@filippog

?!?!

Is this from all nodes? Some subset?

[eevans]

From IRC:

11:05 < urandom> godog: your paste implicates restbase1002 at least, i want to make sure it isn't *just* 1002
...
11:14 < godog> yeah I just saw the same metric from restbase1001 too mobrovac urandom

[gwicke]

@filippog @eevans, those un-prefixed metrics were emitted during startup. See #10.

[filippog]

@gwicke ack, thanks for looking into that