Static labels for scape jobs

[mattkanwisher]

We do discovery of our scrape jobs by using dns_sd_configs, however whats really awkward is we have some static labels that we want to apply to all the machines in the group. We end up having to do relabeling with some strange regexes when all we wanted was just labels. Would you guys be open to taking a pull request in the future that introduces labels at the job level? Or is the are reason you don't do it?

[beorn7]

The domain expert here is @fabxc (currently on vacation).

However, why do you have to do "strange" regexps? .* against a label that is always there (e.g. __name__) should just work. Sure, it's a bit of syntactic noise, but it avoids introducing yet another layer of labeling (there are so many already, and the labels statement in the target_group is arguably historically grown...).

But definitely something to discuss with @fabxc and @juliusv once they are back.

[mattkanwisher]

For example I want 2 static labels one is region: "new york", other is service: "abc service", however there is no way for me to put that in each scrape group. Either I have to do it at the global level or I have to do relabeling. Global won't work, I have many different service groups and regions. Target groups aren't useful if your using DNS or other discovery methods. Not sure what your getting at name, not sure its going to do anything in my usecase, I tried it with no effect. Let me show you the config section

scrape_configs:
  - job_name: clusterxyx-apiname1
    labels:
    target_groups:
    dns_sd_configs:
    - names:
      - _clusterxyz._tcp.apiname1
    scrape_interval:
    scrape_timeout:
    metrics_path: /metrics
    relabel_configs:
    - source_labels:
      - __address__
      target_label: region
      regex: .*\.(\w+)\.test\.somewebsite\.com.*
      replacement: $1
      action: replace
    - source_labels:
      - job
      target_label: service
      regex: (.*)
      replacement: $2
      action: replace

What would be nice is a labels section under the -job_name section instead of relabeling

[matthiasr]

@beorn7 I think this makes sense from a usage perspective. If you have an SD mechanism that can discover everything and knows what is what (e.g. location, other tags) you should be able to transport this through. The "old school" way we currently do is having long lists of jobs, but I'd consider that a workaround.

[beorn7]

@mattkanwisher Just for my understanding: Is your problem that something in your config doesn't work as expected, or are you just dissatisfied with the syntactic noise?

What I meant, there is no need to extract your new label from the regexp. replacement could be static. You could do:

scrape_configs:
  - job_name: clusterxyx-apiname1
    labels:
    target_groups:
    dns_sd_configs:
    - names:
      - _clusterxyz._tcp.apiname1
    scrape_interval:
    scrape_timeout:
    metrics_path: /metrics
    relabel_configs:
    - source_labels:
      - __address__ # always exists
      target_label: region
      regex: .* # always matches
      replacement: new york
      action: replace
    - source_labels:
      - __address__ # always exists
      target_label: service
      regex: .* # always matches
      replacement: abc service
      action: replace

[mattkanwisher]

Nice that cleaned up my config a whole lot. I can live with this, what you just posted. It seems strange to me that their are different options in the hierarchy but it doesn't seem worth much more investment at this point. Thanks!

[juliusv]

@mattkanwisher Yeah, I see your point. The problem is that there are too many levels of granularity where we already allow setting labels (global labels, per-target and per-metric relabeling labels, labels that come from the target, labels that get automatically dervived), and adding more will gradually drive us insane :)

[beorn7]

After discussion with @fabxc , reopening this. We'll rethink if static labels should be possible in SD configs.

[brian-brazil]

I don't think there's any need for additional syntax here, you can already do it with 3 lines of relabel config and in general we don't add configuration for something that's already possible.

[mattkanwisher]

@brian-brazil yeah but prometheus is really unfriendly to configure, like stuffing everything into a relabeling is going to make the system very unusable long term. Things like sharding is now in relabeling also.

[brian-brazil]

stuffing everything into a relabeling is going to make the system very unusable long term.

Our assumption is that anyone who has a system of that scale has configuration management and will use that to manage complexity of their configuration, rather than making prometheus itself hard to configure by offering N different ways to do something that may interact in surprising ways.

Sharding is an advanced use case needed by very few users (3 so far), optimising for it isn't a good investment.

[beorn7]

What gives me an itch is that we have static labels with file-based SD and with static target config, but not with the other SDs. I want it to be a feature consistently there or consistently absent.

[brian-brazil]

I don't think that's inconsistent, it's a feature of the SD per-target rather than the scrape config per-job. Other SDs are free to add in whatever labels they like too based on the SD (thus far the SD methods aren't use consistently enough to be able to pick out useful labels for everyone - this won't be the case with EC2 say where region and availability zone labels could be applied)

[beorn7]

But if you are fine with 3 lines of relabel config, why do we have static labels in static target config? The same method (3 lines of relabel config) could be used there to attach static labels.

[brian-brazil]

Ah now I see what you're saying, the Labels in TargetGroup indeed doesn't fit.

[beorn7]

Yup. We could declare it legacy (with the intention to deprecate and remove it eventually), or we could offer the same everywhere (which might be one level of labeling too many...).

[fabxc]

Then again TargetGroup is how SD hands around labels. Arguably static targets have no chance to receive labels via SD so it might even be justified to have them in static target groups in the config file.

[brian-brazil]

With replacement and regex defaulted for relabelling, it's now much simpler to do static labels so I'd say this is good enough.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.