Add dyno name to host name

[mezis]

Optionally adds the dyno name to the hostname reported to Datadog.

[miketheman]

Hi @mezis! Thanks for the contribution.
I had thought hard about the dyno ID a while back, and had a hard time justifying it's use, because I don't think the user has the ability to affect change on individual dynos.
Can you explain how this would benefit the user?

[mezis]

Can you explain how this would benefit the user?

Mainly by giving the user the ability to see metrics by dyno. By default you'd constantly be averaging out, and could miss outliers — which can well be related to the code one runs, rather than the infrastructure.

[konobi]

Is this usable? I think we're getting bad metrics since we have multiple web workers under the same app, trampling over each other. Host needs to be set so the metrics can be separated or even aggregated properly.

[miketheman]

I've been thinking about this approach, and I don't think it's the best idea. Here's some reasoning:

• The $DYNO variable contains a number, i.e. web.1 or worker.2 - so prepending this to the hostname will drive up the overall host count.
• The worker number will make it harder to combine metrics from a given worker type, as you then have to plot over host + host + host + ...

What about adding a dyno_type tag to the Agent, so all metrics emitted by that dyno get tagged with the worker type?
I put together a POC here: 054f24c

[konobi]

In the datadog ui, there's the option with the requests metric to choose something to group by, and it's always "host", but for this buildpack, it's always blank. That's the "host" metadata that I'm thinking of.

[miketheman]

@konobi I don't think I follow - part of the setup for an app is setting a hostname in the config and it errors out otherwise.

I tried today, and metrics emitted from inside the web worker show a host value in the dropdown.

[konobi]

okay... right now that HEROKU_APP_NAME is the name of the app... but you have multiple instances of that app on different dynos. So instead of host:$HEROKU_APP_NAME it should be something more like host:$HEROKU_APP_NAME-$DYNO_NAME so that we can differentiate between hosts (and also provide totals appropriately). In my case, it's a web process.

[miketheman]

@konobi I understand that - that's why I am proposing instead of munging the host, we add the dyno_type tag, as suggested above: #10 (comment)

Then use can use the intersection of the existing host and dyno_type to scope a query.

[konobi]

k... that seems reasonable, assuming it's UI selectable =0)

[miketheman]

So it turns out that the tags variable from the config file isn't being used by the dogstatsd mechanism, so the fix proposed isn't 100% viable.

I did make a viable workaround as an example (in ruby, other languages have similar env var approaches):

$dyno_type ||= ENV['DYNO'].split('.').first
$statsd.increment('page.views', tags: ["dyno_type:#{$dyno_type}"])

This provides the "drop down" selection to be per-app (aka host) and dyno type.

It's not perfect, but it's better than creating many "virtual" hosts (increases billing, makes it more difficult to maintain dashboards).

I'll continue to explore a better method for this.

[konobi]

any luck?

[miketheman]

I raised this issue with the Agent codebase, and they indicate that this mechanism is unlikely to be modified.

I think it boils down to a couple of points:

• Using the appname.dyno_id approach here, the billable host count increases, as each individual dyno has its own host representation. The complexity of reporting a sum across all dynos with given metric is increased, as the dyno id is a unique host and Datadog doesn't support "wildcard" hostname queries.
  Is this acceptable? In my original response to this PR, I didn't think so, and I still don't think it's a significant benefit.
• Using a runtime-determined set of tags that would need to be added to any calls is less "automatic", as it requires the addition of a tags: ["dyno_type....] added to each call.
  While this could be solved via some fancy meta-programming that I haven't looked for yet, it does allow for good query practices on the Datadog UI, as you now have another axis/parameter to use that isn't hard-coded into the hostname, and allows for extension to other tags.
  It might even encourage further use of the tags array.

If there are other ideas or perspectives that I'm not seeing, I welcome them.

[konobi]

tags seems like a reasonable enough approach to me, though cost at the datadog end isn't nessecarily a factor for us specifically.

[konobi]

Also, is there any documentation on how to configure kafkacat for use with ssl/tls?