How to generate metrics across replicas possibly on different machines

[MatthewScholefield]

I'm trying to figure out if I can generate Prometheus metrics for a service I scale with multiple replicas, each running in isolation (ie. within a docker container). Is this possible?

I know I can just add multiple scrape endpoints within a metric source in Prometheus's config, but I'm unsure if this would introduce errors when multiple machines emit gauge values for the same metric. I'm assuming it might work if I follow the steps mentioned for multiprocess mode, but I had the following concerns:

1. Will multiprocess mode work even across machines (or does, for instance, use pids to check for liveness)?
2. Does the PROMETHEUS_MULTIPROC_DIR folder need to be shared across machines (ie. a mounted volume)?

High level, I feel like it's a common situation to want to emit metrics from a stateless replicated web service so if there's a completely different way most people do this I'd be happy to hear.

[csmarchbanks]

Hello,

I converted this to a discussion instead of an issue as it appears to be a usage question, not a bug/feature request for client_python.

Prometheus will handle metrics across replicas for you by adding an {instance="my-host:8080"} label to each series when ingested. You can then tell the series for each machine apart when querying using the instance label. You can read more about that behavior here: https://prometheus.io/docs/concepts/jobs_instances/.

That said, multiprocess mode does not work across instances in most cases. Your concerns are both valid, as pids are used to identify metrics (and could be re-used by different hosts), and PROMETHEUS_MULTIPROC_DIR would need to be shared and writeable by each machine. I would definitely not recommend trying to use multiprocess mode across machines :).