metrics: prometheus integration road map #27307

Open
stevvooe opened this Issue Oct 11, 2016 · 16 comments

Projects

None yet

7 participants

@stevvooe
Contributor
stevvooe commented Oct 11, 2016 edited

At the Docker Distributed Systems Summit, we discussed the integration of prometheus into docker. We made several clarifications around how to match the two models, both at the engine-level and the swarm mode-level.

Most importantly, we defined the high level steps that need to be taken to achieve a nice result:

  • Integrate a prometheus metrics output for internal behavior of docker itself. Currently, we are interested instrumenting container startup, but may include further metrics. This will help support the Docker Maintainer and Contributor team to make docker better while also allowing use to learn about using prometheus in a production product. (#25820)
  • Expose externally observable containers metrics. This should replace use cases that involve docker stats and external metrics exporters. The focus must be on a stable and scalable schema, while supporting the future goals of application-level metrics. We may have to rely on externally support target discovery, but it may make sense to tackle some of this in the interim.
  • Define cluster-level discovery mechanism to expose topology to prometheus discovery. This will include integration into swarm-mode as well as single node target discovery. We may find that we need to do this in conjunction with externally observable metric.
  • Expose per-container metrics including application-level proxied metrics. This will mean that application and container-level metrics will be served under a single target. No aggregation will be done in the docker engine. Such an integration will be a pass-through proxy, annotated with target-specific data, such as container ids.

cc @juliusv

@juliusv
juliusv commented Oct 11, 2016

Thanks @stevvooe, that's a great summary of our discussion! One question:

This will mean that application and container-level metrics will be served under a single target.

Does this mean you'd want to merge together the internal whitebox metrics from an application's /metrics endpoint together with the externally observable container metrics as supplied by Docker itself? I'd say there's cases where you'd want to scrape one, but not the other (for example, a cluster admin would only want to ingest external metrics), so maybe better to keep it separate? But maybe I'm misunderstanding it.

@brian-brazil

Expose externally observable containers metrics.

This should go somewhere other than /metrics, as otherwise these metrics will take significant resources away from both infrastructure level monitoring and service-level monitoring due to them having to monitor everyone else's machines/containers.

Expose per-container metrics including application-level proxied metrics.

Scraping of applications should happen directly. Application metrics are not container metrics, and similarly container metrics are not application metrics.

Such an integration will be a pass-through proxy, annotated with target-specific data, such as container ids.

This is an anti-pattern in Prometheus. Target labels should come from service discovery and relabelling, not the target itself.

@stevvooe
Contributor

@brian-brazil @juliusv Saying something is an anti-pattern is an anti-pattern. 🐹

Let's try to view this more as a high-level road map than a place for discussion of technical details.

The design discussion needs to really be around how we integrate target discovery to control metadata tagging. If I was not clear above, there will be no tagging if we do any kind of proxying. This will all be done via the target metadata associated with the scrape. While I agree that it would be ideal for applications to be scraped directly, there may be complexities created in that model that can be avoided if the management plane is leveraged.

The options are pretty much as follows:

  1. Have one, gigantic slow metric endpoint that does everything.
  2. Expose internal engine targets with a separate target for each container, along with application metadata. This has the benefit keeping the container performance associated with the application.
  3. Separate out into internal engine, container targets and application targets.

I suspect the right answer is between 2 and 3.

Remember, the goal here is to require almost zero configuration for this to work out of the box. Anything that involves running a specific container on a specific network can start making this more complex for end users.

BUT, let's discuss this in further detail after we've been through steps of exporting engine metrics (step 1 on the road map). I suspect there are some misunderstandings around terminology that will be more clear when we have further experience. Specifically, there are some details around target discovery that I'm a little unclear on.

@juliusv
juliusv commented Oct 12, 2016

@brian-brazil As background information, the motivation for scraping apps through a transparent proxy (with target metadata coming via the SD as target labels, not in the metrics output itself) is that on Docker, people run everything in their own little segmented virtual Docker networks (per service or similar), so Prometheus would be unable to reach apps directly except for the ones in its own Docker network. That transparent proxy would effectively punch a hole through that networking segmentation for the purpose of gathering metrics.

@jimmidyson

This is a great idea - looking forward to it already. In time, feels like it would make sense to be part of OCI spec perhaps?

@brian-brazil

Thanks for the clarification. I'd propose 3). A transparent proxy sounds like a good idea to handle varying network deployments.

I don't think injecting metrics into the application's metrics via the proxy is workable due to collisions, the potential for multiple applications living inside one container and generally getting in the way (consider the blackbox or snmp exporter, container metrics are irrelevant for its typical usage).

Specifically, there are some details around target discovery that I'm a little unclear on.

Anything in particular we can help clarify?

@stevvooe
Contributor

I don't think injecting metrics into the application's metrics via the proxy is workable due to collisions, the potential for multiple applications living inside one container and generally getting in the way (consider the blackbox or snmp exporter, container metrics are irrelevant for its typical usage).

We'll have to try a few things out here. I think we'll know more when we start seeing the volume of container metrics.

It might be good to review these PRs based on metrics endpoint output, rather than purely code. I think it will clear up a lot of confusion and that will be the interface that we standardize upon.

Anything in particular we can help clarify?

I just need to go read the implementation. This one is on me.

@brian-brazil

We'll have to try a few things out here. I think we'll know more when we start seeing the volume of container metrics.

I don't see the volume being a major issue as long as we avoid 1), the challenge is more around semantics and edge cases.

I just need to go read the implementation. This one is on me.

The short version is that you give Prometheus a list of targets, each target having whatever metadata might be useful as a set of key/value pairs. I'm guessing here we'd be talking about a regular poll of the Docker API to get this information. That's then offered up to the user to munge with relabelling.

The main things to do then are determine how/what information is pulled from Docker, and what the example Prometheus configuration looks like for this (there'll likely be a moderate amount of boilerplate, so you want something copy&pasteable).

@thaJeztah thaJeztah added the roadmap label Oct 15, 2016
@jmkgreen

Is there a discussion document that defines the scope of this work that you guys can supply a reference to? Having stumbled upon this, it looks both interesting and potentially very broad.

Some goals and non-goals would be helpful.

@stevvooe
Contributor

@jmkgreen What aren't we covering in the description of this issue? Indeed it is broad. Creating a rigid, structured plan is both a lot of work and won't necessarily provide a better result, especially when there is a feedback component to each stage.

As we take each item here, we will create issues that cover the details of the issue at hand. Scope and goals will be better defined at that time. If we need to adjust scope based on the results of that plan, we will adjust this road map.

Is there something specific missing here? What decision are you trying to make?

@jmkgreen

The original post reads as though there are already a set of intentions in mind. When talking of goals and non-goals, I do not expect a long bullet-list, but would like to know what is to be monitored and who is the expected audience of it's output.

Are we talking operations people monitoring hardware capacity here ("we need to order more machines"), the operating capacity of individual docker containers ("we need to spread the load more" or "we can evacuate these machines for reboot and still cope"), or applications engineers ("we clearly have a memory leak when in production" or "we need to look at the db - it is really slow").

Monitoring is an enormous topic. What are you trying to do here?

@errordeveloper
Contributor
@stevvooe
Contributor

@jmkgreen If you read carefully, your questions are covered in the bullet points in the description. I'll admit, there is a leap in context in favor of brevity. It might help to read up on prometheus to better understand the points.

Let me break this down into simpler bullet points:

  • Docker engine metrics
  • Externally observable container metrics (CPU, memory, etc.)
  • Integrated target discovery (which endpoints to scrape metrics from)
  • Application-level metrics (forward targets for running container applications)

With all of these implemented, one should be able to hit all of the described use cases. However, the goal of this exercise is data flow. How you consume those metrics and who consumes those metrics is really out of scope. That should be up to the operators of the infrastructure.

As you have posed your inquiry, it sounds like you are just critiquing the methodology. If you could expand on why your asking this question or what conclusion you are attempting to make, I may be able to give you a better answer.

@jmkgreen

So this boils down to allowing for discovery of topology and the polling of metrics both within Docker and it's hosted applications? If so, that's a fine summary, and a welcome move. Finally, I assume this will result in something both prometheus and others can work against?

@brian-brazil

Prometheus is an open ecosystem, we've currently parsers in Go and Python for our format that others can use to integrate with whatever they like. For example with our Python client library it's possible to write less than 10 lines of code that can regularly fetch Prometheus-formatted data and push it out to Graphite.

I don't know the full details on the discovery and forwarding, but I'd be very surprised if there's anything that ties it to Prometheus.

@stevvooe
Contributor

@jmkgreen Expaning on, @brian-brazil's response, that is exactly the case. Prometheus was chosen for its format. There may be community work to flow prom data into other systems but filling those gaps will help both the prom community and docker users, alike.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment