Jobs/insrtances documentation disambiguity

[msiebuhr]

When submitting data with job and/or instance-tags set, say foo{job="a.herokuapp.com",instance="b"}, the docs say it will add new keys with exports_-prefix.

But instead it renames the existing keys to something exports_job and exports_instance, which isn't really useful, as grouping metrics from different upstreams by these keys would group a lot of data by the aggregator's name instead of the original job/instance names.

[msiebuhr]

Relevant documentation: http://prometheus.io/docs/concepts/jobs_instances/

[juliusv]

Thank you! This indeed changed in 0.15.0 and it wasn't updated in the docs yet. Here's the PR to fix it: prometheus/docs#148

[msiebuhr]

After thinking it over, I think I'd prefer it be the other way around (i.e. that Prometheus doesn't mess with the existing keys); I have a proxy that ingests the runtime data for a bunch of heroku-machines (see https://github.com/msiebuhr/lumbermill/tree/prometheus). So I have a whole lot of infrastructure that is exposed through a proxy, which means it will (currently) end up like this in prometheus:

some_metric{job="lumbermill",instance="https://...",exported_job="real-server-instance",exported_instance="web.1"}
some_metric{job="lumbermill",instance="https://...",exported_job="real-server-instance",exported_instance="web.2"}
...
# And then some regular job/instance stuff
some_metric{job="system-name",instance="hostname"}

Grouping by job or instance, is in this case meaningless, as most of the metrics will be attributed to the proxy server that is needed to even get the logs to prometheus. Grouping/aggregating by exported_ means that all the "normally" fetched data will show up under one big empty key.

[msiebuhr]

I'm quite curious as to why Prometheus ended up with this behaviour. Any pointers?

[brian-brazil]

Basically if one rogue job starts exporting metrics with labels owned by something else, this keeps it from being a problem.

[msiebuhr]

But you could catch that just as well with just adding tags under exports_, proxy_ or whatever prefix.

For now, I've "fixed" it by adding dyno and app-tags (also maps better to how Heroku names things).

(I have a nagging feeling I'm missing some corner-case that disproves my idea...)

[brian-brazil]

But you could catch that just as well with just adding tags under exports_, proxy_ or whatever prefix.

Unless that's what the rogue job is exporting. We have to design things with all possible reasonable use-cases in mind, and that includes where the person running the Prometheus doesn't trust those writing the monitored jobs not to mess up the labels.

[beorn7]

@msiebuhr There is also the new honor_labels setting in the scrape config (which will be documented soon, prometheus/docs#137 ). That allows you to explicitly whitelist a scrape config to trust the labels.

[msiebuhr]

So (haven't looked at the code) the current behaviour if foo{job="a",exports_job="b"} is to change it to foo{job="proxy_job_name","exports_job="a"}, and entirely discarding the existing exports_job? At least I find that a bit contrived, considering it makes using the two "core" tags useless if the data comes through proxies..

I believe you'd get the same security by plainly setting exports_job on the server, if job is set (overwriting any existing tag). And the jobs-tag would actually be useful.

[msiebuhr]

@beorn7 Great! honor_labels it is.

[beorn7]

@msiebuhr The prefix is exported, and foo{job="a",exported_job="b"} would result in foo{job="proxy_job_name",exported_job="a",exported_exported_job="b"}.

[juliusv]

FTR: All documented under http://prometheus.io/docs/operating/configuration/ and http://prometheus.io/docs/operating/federation/ now btw.

[juliusv]

Anything missing here still or can this be closed?

[msiebuhr]

Looking under federation for how to piece together my Heroku-proxy thingie (indeed, I only looked at the page, now you linked to it).

Besides that nitpick, 👍

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.