Feature request: Define outgoing IP for scrapers

[RichiH]

@brian-brazil As per IRC, it would make some failover scenarios etc simpler to have service IPs which move freely between a pool of machines.

To support this, Prometheus and the exporters need a way to establish all outgoing connections from a configurable IP address.

[RichiH]

@bastischubert

[brian-brazil]

This request is more complicated than it seems, as it end up that you need to support selection by both IP addresses and network interfaces as you don't always know the IP beforehand when DHCP is in play. That gets messy fast across OSes.

I suspect this is too much complexity for Prometheus compared to the gain.

[RichiH]

I don't see DHCP as much of an argument as there should be a CMDB to orchestrate what IP to assign, anyway.

Also, "IP only" falls firmly into 80/20 while avoiding OS-specific ickyness.

[brian-brazil]

I don't see DHCP as much of an argument as there should be a CMDB to orchestrate what IP to assign, anyway.

That's not the case where you don't control the network, such as for a typical DSL setup. If we support one we have to support the other.

I don't think this level of network configuration should be the responsibility of Prometheus.

[RichiH]

In a (non-BGP, or similar) DSL setup, Prometheus needs to use whatever IP the host gets, anyway.

The problem as I see it is that this makes it hard to impossible to detach the IPs to get data from and to collect data with from each other. Those might be in a vastly different security context, which makes separation by IPs and routing the best, and common, choice.

[brian-brazil]

In a (non-BGP, or similar) DSL setup, Prometheus needs to use whatever IP the host gets, anyway.

You're assuming a host only has one IP.

The problem as I see it is that this makes it hard to impossible to detach the IPs to get data from and to collect data with from each other. Those might be in a vastly different security context, which makes separation by IPs and routing the best, and common, choice.

I think a setup where a host/app has multiple IPs to choose from and the choice matters is going to be fairly uncommon. Personally the only one I've seen in many many years is my own esoteric home setup.

Source routing seems pretty niche to me for the sorts of things Prometheus typically handles (especially as we recommend being on the same network as what you monitor), I think it'd be better handled via a proxy or with the various options the kernel networking subsystem provides for controlling choice of source IP, rather than adding that complexity into Prometheus.

[bastischubert]

Hi there,
you're thinking a bit to complex - promtheus shouldn't figure out which ip to use but honor a parameter given by the admin/cmdb/blah...

[RichiH]

I think a setup where a host/app has multiple IPs to choose from and the choice matters is going to be fairly uncommon.

That was not my point, I was referring to a case where blackbox_exporter is used to simulate access from external/internal or otherwise.

Source routing seems pretty niche to me for the sorts of things Prometheus typically handles

Source routing is only an afterthought, initially, I wanted to make the service IP simpler to transfer. Reason being that we're required to have several layers of access lists and firewalling even in outward-facing services. The few IPs which do get access to the target exporters should be as agile as possible in case of system failure or load distribution.

or with the various options the kernel networking subsystem provides for controlling choice of source IP

I couldn't find a clean way to do that, pointers would be appreciated, a configurable option preferred :)

[brian-brazil]

I wanted to make the service IP simpler to transfer.

When I think service IP in terms of Prometheus I think the IPs it's serving on, not what it's connecting out on. That sounds like a VIP for inbound connections.

Reason being that we're required to have several layers of access lists and firewalling even in outward-facing services. The few IPs which do get access to the target exporters should be as agile as possible in case of system failure or load distribution.

This sounds like an organisational rather than technical issue. I'm very wary of adding complexity purely to offer ways to workaround non-technical issues, as such problems are far less bounded than fundamental technical issues.

I couldn't find a clean way to do that, pointers would be appreciated, a configurable option preferred :)

http://lartc.org/howto/lartc.rpdb.html

[RichiH]

When I think service IP in terms of Prometheus I think the IPs it's serving on, not what it's connecting out on. That sounds like a VIP for inbound connections.

That's semantics, but I see your point and definition.

This sounds like an organisational rather than technical issue.

While it's obviously not a purely technical issue, access controls are rather common in a multitude of companies and scenarios. As this issue is common and reasonably well bounded, I still think this request has merit.

[brian-brazil]

While it's obviously not a purely technical issue, access controls are rather common in a multitude of companies and scenarios. As this issue is common and reasonably well bounded, I still think this request has merit.

IP based access controls where you have been granted such a limited number of IPs that you need to engineer systems just to reallocate them dynamically is definitely on the organisational side of things. The more common setups would have you statically whitelist all potential Prometheus server IPs/ranges/networks.

We need to draw a line somewhere, organisationally self-inflicted problems are one place where in my opinion it's reasonable to draw that line. We have to be discriminating in what complexity we take on. While it'd be nice to be able to support all possible use cases, taking on every organisations' individual quirks is unlikely to be a winning strategy for the project as a whole or our users in terms of usability or maintainability.

[RichiH]

Let's remove that argument from the discussion, then. I can see your argument even if I don't fully agree with it without limitation.

I still think this request has merit.

[brian-brazil]

I think this request has little merit for Prometheus, as Prometheus works at layer 7 - not layer 3. There's an argument for the blackbox exporter, but it's to do with routing rather than access control.

[dominikschulz]

I'm sorry if I miss some context from IRC here, also I'm fine with limiting the scope of supported use-cases, but to help my understanding: Would it be more complicated than this commit dominikschulz@a057c03 to implement source address selection?

[brian-brazil]

As discussed above that's not sufficient as it only covers IPs, not interfaces. The question of if this meets the bar to be added remains.

[dominikschulz]

Understood. Just wanted to get some better grip on the issue before commenting. Covering the IP use-case should be easy and more or less trivial (like in the linked commit), but supporting anything else will take considerably more code and platform-dependent quirks.

So you're probably right in keeping this kind of support out.

[brian-brazil]

It looks like we'll not be supporting this in Prometheus, but we can always reopen later if more use cases arise.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.