add possibilty to set server_name through relabel config

[efahale]

Hi,

Proposal

Would it be possible to request that server_name in the tls config can be set through relabel config.
I was told in it is not possible to do this when asked in prometheus-users mailing list.
https://groups.google.com/forum/#!searchin/prometheus-users/efahale%7Csort:date/prometheus-users/Ca4pE5MEC8g/xxJtXg5qAwAJ

This will simplify the prometheus configuration where the certitificate doesn't contain targets IP address.

[brian-brazil]

Can you explain more about why you want this?

[efahale]

Thank you for fast reply, Brian

The use case for this is in kubernetes and I guess other cloud platform where the tls certificate is not issued for the ip address but hostname. if you want to have tls connection to several targets which are using different certificates and you don't want to use the insecure_skip_verify=true then you need to create to job for each target where you set the server_name to hostname.

For example if you have 2 pods and they have certificates issued to alfa.io for the first pod and beta.io for the second. To be able to scrape them I need a configuration like
tls_server_name_example_without_relabel.txt
if you could set server_name through relabel config as you can do with adress, scheme then this would be enough.
tls_server_name_example_with_relabel.txt

[brancz]

I think this can be argumented in a few directions:

1. TLS certificates should have the IP as a Subject Alternative Name, this is more difficult though in a whitebox monitoring setting, as it would mean a new and signed certificate needs to be generated, signed and distributed for each target.

2. Similar to 1 but a single certificate is re-issued every time a group of targets that logically belong to each other changes (I doubt this is a good idea).

3. Allow setting the server_name through relabelling, as service discovery typically knows about grouping of targets which can be used to directly set or derive the server_name.

I think my personal opinion is that we should have the flexibility to configure this via service discovery, even if I understand to some degree that 1 might be have the best security characteristics.

I've tried researching this type of PKI problem a number of times, but was never able to find a clear recommendation for 1 or 2, but I'm more than happy to learn something new 🙂 .

[efahale]

Hi,
would it be ok if I make a pull request for adding the possibilty to set server_name through relabel config ?

[tsaarni]

Using IP address as subject for a certificate is problematic in Kubernetes since pod IPs are ephemeral. Consider an example:

1. Pod A is created and it gets assigned address 10.1.0.24. We issue certificate with SAN IP:10.1.0.24
2. Container in pod A exits and pod is recreated with new IP address 10.1.0.25. New certificate is issued with SAN IP:10.1.0.25.
3. Pod B is created and it could get recycled IP address 10.1.0.24. Certificate with SAN IP:10.1.0.24 is issued again.

Now two valid certificates exist with SAN IP:10.1.0.24 - one that was associated with pod A and second associated with pod B. Identities based on short lived IP address assignments would require very elaborate automation, including realtime support for revoking certificates and verification of revocation status at each time client connects the server. Therefore I would consider any authentication scheme based on short lived IP address to be a bad match from security perspective.

Subject name should be some stable identifier, preferably something provisioned by administrator. Consider another example:

1. Administrator issues certificate with SAN DNS:A
2. Pod A is started. Certificate with SAN DNS:A and associated private key are provisioned to it
3. Administrator provisions Prometheus with relabelling rule and pod label server_name="A"
4. Prometheus establishes TLS connection to pod A using the dynamically allocated IP address but provides SNI server_name "A" based on the relabelling
5. Pod A returns the certificate with SAN DNS:A
6. Prometheus validates that certificate is issued by trusted CA and that it has SAN DNS:A, according to relabelling rule

After TLS handshake there is a proof that Pod A is in possession of private key associated to certificate with SAN DNS:A.