Can't relabel based on instance from pushgateway

[TheTincho]

Hi,

I have set up a pushgateway to collect stats from logs in different instances of the same service. As I want to be able to easily match these data with the node_exporter stats, I am creating a label 'host' from the instance.

This works well for the node_exporter job (using the __address__ meta metric as a source), but I cannot make it work with the pushgateway; I've tried relabel_configs and metric_relabel_configs with the same regex that works for the node_exporter, using the instance metric, but it seems that it is being ignored.

Config snippet:

  - job_name: pushgateway
    metric_relabel_configs:
      - source_labels: ['instance']
        target_label: 'host'
        regex: '^([\w.-]*?)(?:-oam)?(\.[\w.-]*)(?::(\d+))?$'
        replacement: '${1}${2}'

  - job_name: node
    relabel_configs:
      - source_labels: ['__address__']
        target_label: 'host'
        regex: '^([\w.-]*?)(?:-oam)?(\.[\w.-]*)(?::(\d+))?$'
        replacement: '${1}${2}'

[brian-brazil]

First you want the honor_labels setting so that the pusher sets the instance label and then that'll work.

Second without that, you'd want to work with exported_instance.

[TheTincho]

On 02/07/15 15:00, Brian Brazil wrote:

First you want the |honor_labels| setting so that the pusher sets the
|instance label| and then that'll work.

Second without that, you'd want to work with |exported_instance|.

I think this is not the same case you are thinking about. The instance
label is properly set and honoured, even without this flag (which I see
is not present in 0.14.0, which is what I am using now). The problem is
that I don't get the new labels that I want to derive from the instance
name (host, dc).

For example, this is a successful relabel from the host_exporter:

node_memory_MemUsed{dc="cap",host="capXX.YY",instance="capXX-oam.YY:6100",job="node",monitor="ZZ"}

And this is the one that fails:

DocumentStoreEntriesCount{exporter_instance="localhost:6101",exporter_job="pushgateway",instance="capXX.YY:16892",job="JJ",monitor="ZZ"}

As you see, I am missing the host and dc labels.

Martín Ferrari (Tincho)

[beorn7]

@fabxc will know more. I guess this has to do with the exact order of relabelings (wich also might or might not have changed in 0.15.0).

[brian-brazil]

Could it be that instance is only defaulted after the relabel_configs so may not be set yet?

[beorn7]

But in this case, instance is set by the pushgateway already, isn't it?

[brian-brazil]

Without the honor_labels instance would be that of the pgw - maybe it doesn't match the regex?

[beorn7]

According to @TheTincho 's example, the instance label set on the PGW is instance="capXX.YY:16892" - which looks sane to me...

[brian-brazil]

Hmm, that's not the right behaviour at all. I suspect the issue is you're not using a new enough prometheus - try a build from head.

[fabxc]

0.14.0 does not have honour_labels. It also does not have the detection of superfluous config params in YAML configs. So your metric_relabel_configs is discarded silently.

As regular relabel_configs are applied to the static label sets of targets, there is nothing happening here at all as instance is not a static label of the PGW.

What you have should work with 0.15.0rc1.

[TheTincho]

On 02/07/15 18:19, Fabian Reinartz wrote:

0.14.0 does not have |honour_labels|. It also does not have the
detection of superfluous config params in YAML configs. So your
|metric_relabel_configs| is discarded silently.

As regular |relabel_configs| are applied to the static label sets of
targets, there is nothing happening here at all as |instance| is not a
static label of the PGW.

What you have should work with 0.15.0rc1.

To answer all comments at once:

I am setting the 'instance' label in the PGW, via PUT
metrics/job/JJ/instance/hostname -f:16892, so it is a static label.

I tried using relabel_config instead of metric_relabel_configs, and no
change. So I tried replacing all the rules with a dummy one:

  - source_labels: ['component']
    target_label: 'foo'
    regex: '(.*)'
    replacement: '${1}'

And this also gets ignored silently.. Both with 0.14.0 and with
0.15.0rc1, so either I am making some big mistake or relabelling does
not work at all with the PGW, which does not make any sense to me...

Martín Ferrari (Tincho)

[fabxc]

It might be a static label for you but for Prometheus it's not. Static labels are those associated with scrape targets, which is PGW itself in your case and none of the instances pushing to it.

What is the component label in your case? I cannot find it in the examples above.
To clarify: did the instance relabeling rule in metric_relabel_rules with 0.15.0rc1 also fail?

[TheTincho]

Sorry for the silence. I will try to reproduce this in a minimal test and report back.

[TheTincho]

So, I made a test. First with 0.14.0 and then with 0.15.0rc1.

The test configuration is as follows:

  - job_name: 'pushgateway'
    target_groups:
      - targets: ['myserver:9092']
    relabel_configs:
      - source_labels: ['foo']
        target_label: 'bar'
        regex: '(.*)'
        replacement: '${1}'

And the data in the PGW:

$ GET http://myserver:9092/metrics|grep ^up
up{foo="42",instance="myserver:6001",job="test"} 1

I can confirm that there is an issue in both versions:

0.15.0rc1:

$ prometheus-cli -server=http://localhost:9094 query 'up{foo="42"}' 
"up{exported_instance=""myserver:6001"", exported_job=""test"", foo=""42"", instance=""myserver:9092"", job=""pushgateway""}";1;1436322242.335

0.14.0 (copy&paste from the webpage, as I learn now prom-cli does not work with this version!)

up{exporter_instance="myserver:9091",exporter_job="pushgateway",foo="42",instance="myserver:6001",job="test"}   1

You can see that in neither case the bar label is created.

To double-check, I performed similar tests with the node_exporter. As expected,
copying the address label worked, but my surprise was when I could not make
it work with a random other label!

Configuration:

  - job_name: 'node'
    target_groups:
      - targets: ['myserver:9100']
      - targets: ['anotherserver:9100']
    relabel_configs:
      - source_labels: ['device']
        target_label: 'iface'
        regex: '(.*)'
        replacement: '${1}'

And the result (0.15.0rc1):

$ prometheus-cli -server=http://localhost:9094 query 'node_network_receive_bytes{job="node"}' |sort
"node_network_receive_bytes{device=""eth0"", instance=""anotherserver:9100"", job=""node""}";493905253;1436323179.537
"node_network_receive_bytes{device=""eth0"", instance=""myserver:9100"", job=""node""}";69888967833;1436323179.537
"node_network_receive_bytes{device=""lo"", instance=""anotherserver:9100"", job=""node""}";0;1436323179.537
"node_network_receive_bytes{device=""lo"", instance=""myserver:9100"", job=""node""}";3923313234;1436323179.537

So... Am I doing something wrong here?

[fabxc]

When a new target is discovered it comes with a set of labels that are assigned to all metrics scraped from this target. relabel_configs work on those labels. Per-metric labels that then come with metrics from this target are not affected. This is where metric_relabel_configs should help, which works on labels of the final metrics.

For PGW your target is PGW itself, the instances that pushed to it are entirely unknown to Prometheus. Based on that knowledge the above applies.

[TheTincho]

OK, now I understand my mistake: I misunderstood what relabel_configs was doing, and more confused with 0.14 not supporting metric_relabel_configs and failing silently. I'm sorry about the noise.

Closing this now.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.