Please implement scraping via HTTPS (SSL/TLS)

[stapelberg]

For a small distributed application that I intend to monitor, we’re using HTTPS (with real certificates, so certificate verification is not an issue) and basic authentication instead of a separate private network, mostly because the setup should stay simple and the application runs on multiple different servers, distributed over the internet.

Therefore, could you please add a configuration option to the JobConfig proto message? I’m using SRV record resolving, so I’d like to see e.g. a schema key which defaults to http, but can be set to https.

[beorn7]

Makes sense. Thanks for the input.

[juliusv]

@stapelberg I guess then you'll also need to be able to configure the username/password for the basic auth, so that the final URL will be something like <scheme>://<username>:<password>@<SRV host:port>/<metrics path>?

[stapelberg]

Indeed, yes. For now, I’ve just set up a couple of static targets, but being able to specify userinfo and scheme would be great :).

[juliusv]

Cool, already started coding it. There are just some remaining code hygiene issues I need to figure out before doing a PR.

[m13]

+1
(A document about best security practices should be also amazing)

[juliusv]

Just to give an update, I added TLS and basic auth functionality in this branch: https://github.com/prometheus/prometheus/commits/scheme-and-auth

But to properly test it (disabling certificate checks only during tests, etc.), I think there's some more refactoring to do in how per-job configuration (auth, scheme, disabling TLS checks, ...) is propagated to the targets. I can't spend the time right now, but might later. That's why I'm not opening a PR for that yet.

Anyways, if you're interested, you might want to give this a spin. I hope the changes in the config/config.proto are self-documenting.

[stapelberg]

From my POV, this issue is done. Prometheus 0.14.0 allows me to use the following config:

- job_name: robustirc
  scheme: https
  basic_auth:
    username: 'robustirc'
    password: 'secret'
  dns_sd_configs:
  - names:
    - '_robustirc._tcp.robustirc.net'
  relabel_configs:
  - source_labels: ['__address__']
    regex:         '(.+)\.robustirc\.net:[0-9]+'
    target_label:  'instance'
    replacement:   '$1'

[juliusv]

Indeed, thanks for pointing that out! Closing this issue.

[fnkr]

Here is another example with a static scrape config:

# https://prometheus:foo@myservice.example.com/prometheus_exporter
scrape_configs:
  - job_name: My Service
    scrape_interval: 1s
    metrics_path: /prometheus_exporter
    scheme: https
    basic_auth:
      username: prometheus
      password: foo
    static_configs:
      - targets: ['myservice.example.com']