Static config targets don't allow '/' characters

[hantuzun]

What did you do?
I tried to monitor a service behind my proxy. The service's url has a slash such as: proxy/my_service

What did you expect to see?
I expected Prometheus to start monitoring that url.

What did you see instead? Under which circumstances?
Prometheus has crashed while loading the config, complaining that the target url has a slash:

level=error msg="Error loading config: couldn't load configuration (-config.file=/etc/prometheus/prometheus.yml): \"proxy/my_service\" is not a valid hostname" source="main.go:159"

Environment

• Prometheus version:

v1.6.1 but the behavior is the same for master head at the moment.

• Prometheus configuration file:

scrape_configs:
  - job_name: my_service
    static_configs:
      - targets:
          - proxy/my_service

Comments
The control for forward slashes in static config targets are made here: config. CheckTargetAddress

I guess this check is for eliminating trailing slashes, /metrics endpoints, and protocols such as http:// in target urls. I'm curious if there're other reason to reject all service urls with slashes in them.

We would certainly not like to proxy subdomains to the current urls for many reasons.

[juliusv]

The target itself indeed should only contain the host/IP and optionally the port. You can define the path to scrape via the metrics_path scrape config option. See https://prometheus.io/docs/operating/configuration/#<scrape_config>.

[hantuzun]

Hi @juliusv, thanks for looking into the issue.

We have a number of applications to monitor and they share the same host name with different paths. Their URLs are like proxy/service1 and proxy/service2.

However, setting metrics_path at scrape_config level sets the metrics path for all such static configs, and all of their targets, therefore not a suitable solution for this case.

[juliusv]

I see. One option would be to use separate scrape config sections, the other option would be to set the __metrics_path__ meta label to the path you want to scrape for each target (either the service discovery mechanism would have to provide this info somehow, or you provide it manually if you use static configuration of targets). You can also use target relabeling to get that info from potentially some other existing target label that has that information and copy it into the __metrics_path__ label during the relabeling phase.

[hantuzun]

Hi,

Yes, I guess it wasn't possible to add a number of scrape configs since scrape configs are defined at config root via its key. But modifying __metrics_path__ worked for me! Thank you for pointing it out. I'm pasting an example config for reference:

---

scrape_configs:
  - job_name: service1
    static_configs:
      - targets:
          - host
    relabel_configs:
      - source_labels:
          - __metrics_path__
        action: replace
        target_label: __metrics_path__
        replacement: /path1/metrics
  - job_name: service2
    static_configs:
      - targets:
          - host
    relabel_configs:
      - source_labels:
          - __metrics_path__
        action: replace
        target_label: __metrics_path__
        replacement: /path2/metrics

Update: I had to change metrics_path for jobs. Much simpler:

---
scrape_configs:
  - job_name: service1
    static_configs:
      - targets:
          - host
    metrics_path: /path1/metrics
  - job_name: service2
    static_configs:
      - targets:
          - host
    metrics_path: /path2/metrics

[zhoulouzi]

I got the same questions。some hosts was behind the proxy， but i canot access the endpoint directly 。 At first ，I do not want to have so many job__name, too difficult to manage, but finally I give each job with a set of the same label, so that it seem to be a group of jobs. Not a way to approach

---

scrape_configs:

• job_name: service1
  static_configs:
  • targets:
    • host
      labels:
      job_group: 'XXX'
      metrics_path: /path1/metrics
• job_name: service2
  static_configs:
  • targets:
    • host
      labels:
      job_group: 'XXX'
      metrics_path: /path2/metrics

[javamuc]

I'd suggest to reopen this issue and allow '/' in the target host name. It makes the configuration unreadable if I have to provide for each service behind a proxy with it's own job definition in the prometheus config.

[juliusv]

@javamuc You can set the metrics path for each target of a scrape config individually by setting the __metrics_path__ meta label.

[javamuc]

I see but I still can't seem to work it out.

What I would like to have is this:

scrape_configs:
  - job_name: dev-space
    static_configs:
      - targets:
          - host1/service1
          - host1/service2
          - host1/service3
    metrics_path: /metrics
  - job_name: live-space
    static_configs:
      - targets:
          - host2/service1
          - host2/service2
          - host2/service3
    metrics_path: /metrics

The workaround I see, and what you recommend if I understand your correctly, is in the comment above: "antuzun commented on May 10, 2017 ", but that does not solve that I have to have a job for every single service.
I think I'm missing something, could you post a short example how I could achieve the configuration I've given with only two jobs by using the __metrics_path__ meta label?

[jalberto]

prometheus allows multiple target with multiple ports & hosts, to not allow multiple paths is just inconsistent.

Yes this can be done with relabeling, or with mutliple jobs, but is just cumbersome and confusing

[propertone]

I stumbled upon this issue as well. In case this is still relevant to anyone, you can work around this issue with a single job using the following relabel config:

- job_name: scrape
  static_configs:
    - targets:
      - host1/service1
      - host1/service2
      - host2/service1
      - host2/service2
  relabel_configs:
    - source_labels: [__address__]
      target_label: __metrics_path__
      regex: '(.*)/(.*)'
      replacement: '/$2/metrics'
    - source_labels: [__address__]
      target_label: instance
    - source_labels: [__address__]
      regex: '(.*)/(.*)'
      replacement: '$1'
      target_label: __address__

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.